Re: [vserver] hybrid zfs pools as iSCSI targets for vserver

From: Gordan Bobic <gordan_at_bobich.net>
Date: Sat 06 Aug 2011 - 21:59:21 BST
Message-ID: <4E3DAB29.4070001@bobich.net>

On 08/06/2011 09:51 PM, John A. Sullivan III wrote:
> On Sat, 2011-08-06 at 21:37 +0100, Gordan Bobic wrote:
>> On 08/06/2011 09:30 PM, John A. Sullivan III wrote:
>>> On Sat, 2011-08-06 at 21:40 +0200, Eugen Leitl wrote:
>>>> I've recently figured out how to make low-end hardware (e.g. HP N36L)
>>>> work well as zfs hybrid pools. The system (Nexenta Core + napp-it)
>>>> exports the zfs pools as CIFS, NFS or iSCSI (Comstar).
>>>>
>>>> 1) is this a good idea?
>>>>
>>>> 2) any of you are running vserver guests on iSCSI targets? Happy with it?
>>>>
>>> Yes, we have been using iSCSI to hold vserver guests for a couple of
>>> years now and are generally unhappy with it. Besides our general
>>> distress at Nexenta, there is the constraint of the Linux file system.
>>>
>>> Someone please correct me if I'm wrong because this is a big problem for
>>> us. As far as I know, Linux file system block size cannot exceed the
>>> maximum memory page size and is limited to no more than 4KB.
>>
>> I'm pretty sure it is _only_ limited by memory page size, since I'm
>> pretty sure I remember that 8KB blocks were available on SPARC.
> Yes, or for example, Oracle can write directly bypassing the file system
> and thus works very well with iSCSI by setting very large block sizes.

That sounds very much like your problem is in the FS layer rather than
iSCSI. Unless you are saying that it insists on ACK-ing per-operation.
But the FS flush will typically commit in large batches rather than
block-by-block, unless your every write is fsync()-ed.

>>> iSCSI
>>> appears to acknowledge every individual block that is sent. That means
>>> the most data one can stream without an ACK is 4KB. That means the
>>> throughput is limited by the latency of the network rather than the
>>> bandwidth.
>>
>> Hmm, buffering in the FS shouldn't be dependant on the block layer
>> immediately acknowledging unless you are issuing fsync()/barriers. What
>> FS are you using on top of the iSCSI block device and is your
>> application fsync() heavy?
> The application is for standard file service and we are not using
> barriers. We have tried using the device as disk, as part of LVM, as
> part of a RAID device, as part of dm-multipath multi-bus. Pretty much
> the same results across the board. We could produce higher aggregate
> throughput with RAID and multibus by multiplexing several individual
> streams but even then only after changing from the default CFQ scheduler
> to noop (which I suppose makes sense when writing to a SAN). Individual
> streams are still limited by latency.

I find that the deadline scheduler works best for most things.

> When we are drawing from cache, our systems fly but anything that
> touches disk is painfully slow - John

I presume you have disabled things like atime. What kernel are you
running and what fs? I have found that moving from 2.6.29 to 2.6.38 with
ext2 made a very noticeable difference to disk I/O performance on slow
media (SD cards in my case). ext4 (without the journal) is even better.
Could it be that you are suffering from a kernel "feature" that has
since been fixed?

Gordan
Received on Sat Aug 6 21:59:31 2011

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Sat 06 Aug 2011 - 21:59:31 BST by hypermail 2.1.8