[Novalug] benchmarks and iSCSI

[John Franklin]

On Wed, Feb 06, 2008 at 02:31:56PM -0500, Miguel Gonzalez Castaños wrote:
> John Franklin escribió:
> >1900MB/s?  How are you connecting the iSCSI drive?  100Base-T tops out 
> >at 12MB/s, GigE could do about 100-120MB/s (reality: less). 10GigE 
> >will get you (best, utopian case) 1000-1200MB/s.  How are you getting 
> >1900MB/s?  It sounds like you've got some significant caching on the 
> >client side polluting your results.
> >
> >34MB/s reads sounds like what you should be seeing over GigE, 
> >untuned.  You should be able to boost it some with Jumbo frames, 
> >sysctl net.* tuning and some NIC interrupt tuning.  Also, change the 
> >block size to 1MB.  IIRC, VMFS3 uses 1MB blocks.
> I'm using Gigabit and of course there should be some sort of caching.
> 
> I know about jumbo frames but can you point me to any documentation of 
> how to tune sysctl settings and some NIC interrupts?
> 
> When you are talking about the block size to 1 MB, what you are talking 
> about? The dd test? the filesystem?
> 
> Finally, I'm using VMware. Some people say that the best setting is 
> using iSCSI under Windows and save the virtual hard drives in the NTFS 
> filesystem. I'm trying to compare this configuration with real iSCSI to 
> see what is better. (I have Vmware with Windows as a host OS but I'm 
> using Debian as a guest).

Let's backup for a minute and define the environment.  You've got a
VMware host system (let's call it vmhost1), and a storage server (let's
call it vmnas1), and one or more VMs (let's call them vmguest1,
vmguest2, ... ).

vmhost1 is a Windows machine with a fair bit of storage connected to it.

vmnas1 is a ??? box exporting a block device (the storage) as an iSCSI
device which vmhost1 would like to mount and format NTFS.  vmguest1 and
friends will then create their virtual disks as files on the iSCSI NTFS
file system.

For any given configuration, an iSCSI disk will never be faster than
local attached storage.  The GigE network will limit transfers to about
100MB/s max.  Compared to SCSI (320MB/s) or SATA II (300MB/s), it just
can't compete.

This doesn't mean iSCSI is worthless, especially in VM environments.
Xen and VMWare ESX both support live migrations of VMs from one server
to another.  That is vmguest1 running on vmhost1 can be moved to vmhost2
without shutting down vmguest1.  vmhost1 and vmhost2 manage this by
keeping the virtual drives and a copy of the memory on storage they've
both mounted.  The entire hand-off process can take several seconds, but
vmguest1 keeps running on vmhost1 during most of it until the moment of
transition, so the process to a user is a fraction of a second and
honestly looks like a network hiccup.

Typically, this shared mount is an NFS share, but it can be an iSCSI
device mounted by both vmhost1 and vmhost2 formatted with a filesystem
that supports multiple simultaneous mounts.  VMWare's VMFS3 is one such
filesystem.  Because VMFS3 is built for VM storage, and the virtual
disks will always be very large, VMFS3 uses a 1MB block size, hence the
suggestion of 1MB blocks.

Regarding the tuning HOWTOs, I can't help at all with Windows NIC
tuning.  What kind of box is vmnas1?  If it is an appliance like NetApp
or EMC would sell, contact the vendor. If it's a Linux box, do a Google
search for "linux net sysctl." [1] is one of the better hits I get from
that search.

[1] http://dsd.lbl.gov/TCP-tuning/linux.html

jf
-- 
John Franklin
franklin at elfie.org
ICBM: N37 12'54", W80 27'14" Z+2100'
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://calypso.tux.org/pipermail/novalug/attachments/20080206/55e55077/attachment.pgp