About this list Date view Thread view Subject view Author view Attachment view

From: Yann Dupont (Yann.Dupont_at_univ-nantes.fr)
Date: Wed 14 Sep 2005 - 09:09:43 BST

I cross post to linux-vserver mailing list where I submitted my problem
last week.
and I posted in text + HTML because of the ascii art...

Horms a écrit :

>On Tue, Sep 13, 2005 at 11:04:42AM +0200, Yann Dupont wrote:

>>I there a way to use the postrouting nat rule on a director ??
>No, I do not believe that this is possible without modifying lvs,
>which incidently should be easy enough.
Just a matter of time :-)

>>what I want to do is to dynamically change the source of the packet
>>before it reaches the realserver.
>>I use a masquerading topology.
>>Long explanation (don't read this if you're not interested)
>>I have setup 2 directors ;
>>I have thoses services on the director : Webmail, Ldap, imap/pop ... All
>>thoses services are served by 4 realservers:
>>4 for webmail, 4 for ldap, 4 for imap.
>>All my realservers have 2 network interface : 1 for the service of
>>requests coming from ultramonkey, with an ip rule where default gw is
>>LVS, 1 for administrative (ssh) and requests to other services provided
>>on my intranet (default rule).
>>The problem is coming from the fact that my realservers, are, in
>>reality, vservers (http://linux-vserver.org/), all hosted on 4 physical
>>This configuration works very well, except when I re-enter in the
>>Supose this case:
>>I have connection to Webmail, which in turn needs to ask an ldap server.
>>1) connection to webmail via ultramonkey, realserver Webmail-A is
>>choosen. connexion go to eth0. (Webmail-A is a vserver on host A). reply
>>go through the LVS. OK.
>>2) webmail-A need to ask the ldap. Request go from eth1, and reach the
>>VIP of the director. OK
>>case 3a) director choose Ldap-B or C or D. Ldap request go from eth0 on
>>vserver, and is answerd there, via LVS. All is fine.
>>case 3b) no luck, director choose Ldap-A. Ldap-A is hosted on the same
>>physical server. Ldap-A see packets from Webmail-A to Ldap-A. Ldap-A has
>>a local route (rule 0, local table). He will reply DIRECTLY to
>>Webmail-A, WITHOUT passing by director...
>>This is my problem.

>This problem has cropped up a few times recently, and I don't think that
>anyone has a good solution. That is, if you have a VIP, then the
>real server's servicing that VIP will always connect to themselves
>when accessing the VIP.
In my case directors are separate machines.
I have a total of 6 machines :
2 for directors,
4 for real servers.
This is not exactly the same case as previous post posted recently, or,
at least it's not what I understood. I wasn't clear about this sorry.

Let's try ascii art...

Some client
   | +--------------------------------------------------------+
   | | |
   V V |,389 and others :) |
+----------------+ |
|active director | (x2 -> heartbeat) |
+----------------+LVS-NAT | |
 / | |
/ 192.168.11.xx side | |
| | |
V V |
+-----------+ +-----------+ |
| |Webmail-A .80 (vserver) Webmail-B .81 | | |
| HostA |Ldap-A .85 (vserver) Ldap-B 86 | Host B | |
| |... and others vservers... | | |
+-----------+ 172.20.9.xx side +-----------+ |
 \ / |
  \+->Router<-------------------------------------------+ |
       | |
       | Our intranet |

All the vservers have two IP : & for webmail-A
for exemple

The general case is working well.
The only issue is when a vserver Hosted on A (Webmail-A) make a request
to (VIP of the director)
and the director send the request on Ldap-A (wich is also located on host A)
The request seems local and didn't get by director, so the packet is

But you're right the issue is more or less the same than others posts.

>One way around this, is to have separate VIP's and separate real-servers
>for each service. So Webmail-A and Webmail-B would be different machines
>to Ldap-A and Ldap-B. With LVS-NAT this works as log as Webmail-X and
>Ldab-X are on separate subnets, although they can use the
I fail to see how this can solve the problem...

>same linux director. For LVS-DR Webmail-X and Ldab-X can be on the
>same subnet and share the same linux director.
DR is not an option I consider for the moment... Some directors have
public adresses and provide public services,
Direct replies won't be routed

>If you don't want to add extra machines, you could contrive some setup
>that works with virtual machines, like qemu, xen, user-mode linux,
>vmware, etc...
Well adding more machines isn't a problem,
BTW, my directors ARE already hosted on xen hosts... :-)

The picture was as simplification of the real setup.

I have a total of 2x4 directors (1 for each network served), all packed
on 2 physical
machines with wen :)

And my realservers (the vservers) are already capable of reaching the
good director.
The only case to treat is when a request come from the realservers...

All I want to do is keeping the things as simple as possible...

For the moment I've tried lots of things...
The answer for the moment is :
1) Doing a SNAT on PREROUTING on host A when the packet comes from src
172.20.9.xx -> change it to, say, src 192.168.11.xx
1b) Doing a DNAT on POSTROUTING on host A to dst 192.168.11.xx ->
This way local tables are bypassed.
Unfortunately, PREROUTING only do DNAT and POSTROUTING only do SNAT :(
Maybe it's time to post on netfilter too...

2) Doing SNAT on POSTROUTING on the director,to handle this case. You
just replied to this...

3) Putting a XEN guest somewhere
    a) 1 on HOST A, 1 on HOSTB, just to do handle the case and doing
NAT; But this forces to ALSO use XEN for the linux host that
        handle the vservers, this is not something I want to do for
    b) Putting one more Xen dom_U on the physical machines that hosts
the directors (they already have 4 guest running). Yes this is probably the
        best choice.

4) Maybe investigate more time in NGNET features of vservers, where
networking is now virtualized . This could probably solve my problem
but this feature is still experimental if I remember well.

So I'll probably go for 3) Now. But this seems a litle bit overkill. I
thought there was a better & simpler way to do this.

Thanks for your answer.

Yann Dupont, Cri de l'université de Nantes
Tel: - Fax: - Yann.Dupont_at_univ-nantes.fr

Vserver mailing list

About this list Date view Thread view Subject view Author view Attachment view
[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Wed 14 Sep 2005 - 09:10:05 BST by hypermail 2.1.3