First of all: I've done my research in the eMail Archives before deciding to disturb in here. Maybe I'm not googling for the perfect keyword set. Anyway, here's my question:
I have: * a PowerEdge 2950 with two nics with 4 ports total (e1000 and broadcom). * a MD3000i PowerVault * Vmware Server running on the poweredge.
I'm storing virtual machines in /vmstore (a big partition on the poweredge's own RAID-5 virtual disk). The local storage is for the OS and applications. Not the data.
So, the obvious question here: I want to store the data in the SAN. Should I get my sessions running in the host, or inside each virtual machine?
Also, is it OK to use multi path (dual path) using one broadcom port, and one e1000 port (instead of using the two ports of the same nic)?
Any hints? Suggestions?
Thanks a LOT. I'm very new to this, but I'm doing my best effort.
Arturo 'Buanzo' Busleiman wrote: > So, the obvious question here: I want to store the data in the SAN. > Should I get my sessions running in the host, or inside each virtual > machine?
If this is not the correct group to ask this question, I'd gladly accept suggestions for other groups! :)
On Wed, May 28, 2008 at 01:15:36PM -0300, Arturo 'Buanzo' Busleiman wrote:
> Arturo 'Buanzo' Busleiman wrote: > > So, the obvious question here: I want to store the data in the SAN. > > Should I get my sessions running in the host, or inside each virtual > > machine? > If this is not the correct group to ask this question, I'd gladly accept > suggestions for other groups! :)
I am not sure how you are partitioning your space. Does each guest have an iSCSI target (or LUN) assigned to it? Or is it one big drive that they run from? Also are you envisioning using this with LiveMigration (or whatever it is called with your virtualization system)?
On May 28, 2:45 pm, Konrad Rzeszutek <kon...@virtualiron.com> wrote: > I am not sure how you are partitioning your space. Does each guest > have an iSCSI target (or LUN) assigned to it? Or is it one big > drive that they run from? Also are you envisioning using this > with LiveMigration (or whatever it is called with your virtualization > system)?
I'm using Vmware-Server (not ESX, just the free one).
The guests themselves (the disk where the OS is installed) are stored as vmdk's on a local folder.
I want to provide application storage for each virtual machine, no shared storage. I have 1.6TB total capacity, and plan on giving each guest as much raid-5 storage space as they need.
The iscsiadm discovery on my Host reports all available targets, over both interfaces (broadcom and intel).
So, basicly, I have these doubts / options:
1) Login to each target on the host, and add raw disk access to the guests to those host-devices. 2) Don't use open-iscsi on the host, but use it on each guest to connect to the targets.
And the main doubt: how does link aggregation / dualpath fit into those options?
Also, i find this error:
root@VGEVS02:~# iscsiadm -m node -L all Login session [iface: default, target: iqn.1984-05.com.dell:powervault.6001e4f0004326c100000000482127e3, portal: 192.168.130.102,3260] Login session [iface: default, target: iqn.1984-05.com.dell:powervault.6001e4f0004326c100000000482127e3, portal: fe80:0000:0000:0000:021e:4fff:fe43:26c3,3260] iscsiadm: initiator reported error (4 - encountered connection failure) iscsiadm: Could not log into all portals. Err 107.
On Thu, May 29, 2008 at 02:35:28PM -0300, Arturo 'Buanzo' Busleiman wrote:
> On May 28, 2:45 pm, Konrad Rzeszutek <kon...@virtualiron.com> wrote: > > I am not sure how you are partitioning your space. Does each guest > > have an iSCSI target (or LUN) assigned to it? Or is it one big > > drive that they run from? Also are you envisioning using this > > with LiveMigration (or whatever it is called with your virtualization > > system)?
> I'm using Vmware-Server (not ESX, just the free one).
> The guests themselves (the disk where the OS is installed) are stored as > vmdk's on a local folder.
> I want to provide application storage for each virtual machine, no > shared storage. I have 1.6TB total capacity, and plan on giving each > guest as much raid-5 storage space as they need.
> The iscsiadm discovery on my Host reports all available targets, over > both interfaces (broadcom and intel).
> So, basicly, I have these doubts / options:
> 1) Login to each target on the host, and add raw disk access to the > guests to those host-devices. > 2) Don't use open-iscsi on the host, but use it on each guest to connect > to the targets.
If you run iSCSI on each guests you end up with overhead. Each guests will have to do its own iSCSI packet assembling/disassembling, along with doing socket operations (TCP, IP assembling) and your target will X-Guests connections. Each guest would need to run the multipath suite which puts I/O on the connection every 40 seconds (or less if a failure has occurred).
If on the other hand you make the connection on your host, setup multipath there, create LVMs and assign them to each your guests you have: - less overhead (one OS doing the iSCSI packet assembling/disassembling), TCP/IP assembling. - one connection to the target. You can even purchase extra two NICs and create your own subnet for them and the target so that there is no traffic there (except iSCSI). - one machine running multipath and you can make it queue I/O from place if the network goes down. This will block the guests (you might need to change the SCSI timeout in the guests - no idea what registry key you need to change for this in Windows). - One place to zone out your huge capacity and you can resize them as you see fit from one place (using LVMs for guests and you can re-size them).
> And the main doubt: how does link aggregation / dualpath fit into those > options?
I can't give you an opinion about link aggregation as I don't have that much experience in this field.
But in regards to multipath you are better of doing it on your host than on the guest.
> Also, i find this error:
> root@VGEVS02:~# iscsiadm -m node -L all > Login session [iface: default, target: > iqn.1984-05.com.dell:powervault.6001e4f0004326c100000000482127e3, > portal: 192.168.130.102,3260] > Login session [iface: default, target: > iqn.1984-05.com.dell:powervault.6001e4f0004326c100000000482127e3, > portal: fe80:0000:0000:0000:021e:4fff:fe43:26c3,3260] > iscsiadm: initiator reported error (4 - encountered connection failure) > iscsiadm: Could not log into all portals. Err 107.
Did you configure your ethX to use IPV6? The second target IP is in IPv6 format.
> I'm using crossover cables.
No switch? Then link-aggregation wouldn't matter I would think (since the ARP requests aren't going to a switch).
Arturo 'Buanzo' Busleiman wrote: > On May 28, 2:45 pm, Konrad Rzeszutek <kon...@virtualiron.com> wrote: > > I am not sure how you are partitioning your space. Does each guest > > have an iSCSI target (or LUN) assigned to it? Or is it one big > > drive that they run from? Also are you envisioning using this > > with LiveMigration (or whatever it is called with your virtualization > > system)?
> I'm using Vmware-Server (not ESX, just the free one).
> The guests themselves (the disk where the OS is installed) are stored as > vmdk's on a local folder.
> I want to provide application storage for each virtual machine, no > shared storage. I have 1.6TB total capacity, and plan on giving each > guest as much raid-5 storage space as they need.
> The iscsiadm discovery on my Host reports all available targets, over > both interfaces (broadcom and intel).
> So, basicly, I have these doubts / options:
> 1) Login to each target on the host, and add raw disk access to the > guests to those host-devices. > 2) Don't use open-iscsi on the host, but use it on each guest to connect > to the targets.
> And the main doubt: how does link aggregation / dualpath fit into those > options?
> Also, i find this error:
> root@VGEVS02:~# iscsiadm -m node -L all > Login session [iface: default, target: > iqn.1984-05.com.dell:powervault.6001e4f0004326c100000000482127e3, > portal: 192.168.130.102,3260] > Login session [iface: default, target: > iqn.1984-05.com.dell:powervault.6001e4f0004326c100000000482127e3, > portal: fe80:0000:0000:0000:021e:4fff:fe43:26c3,3260] > iscsiadm: initiator reported error (4 - encountered connection failure) > iscsiadm: Could not log into all portals. Err 107.
open-iscsi does not really support link local ipv6 adders yet. There is a way around this by setting up a iface for the net inerface (ethX) that it is local to then binding it, but it is a little complicated to do.
On May 30, 10:29 am, Konrad Rzeszutek <kon...@virtualiron.com> wrote: > If you run iSCSI on each guests you end up with overhead.
I decided the same. So I'm running iscsi on the host, and via /dev/disk/by-path/whatever provide the guests with SAN storage.
Bad thing there, for each virtual disk of the san, I get two /dev entries, so I'm wondering how to setup the multipath over those two /dev/disk/by-path entries (one over each controller).
I also noticed the IPv^ thing, so I ended up specifying the IP by hand, and using to iscsiadm commands for each discovery / session initiation. Also, as you said, link aggregation makes no sense over crossover and two different interfaces.
What I'm NOT doing, is LVM. I wanted to go one layer at a time, and adding LVM was too much for my limited time in here.
So, currently, I have two remaining issues:
1) setup multipath 2) **URGENT**: I've added a second virtual disk and mapped it to my host (SAN is an MD3000i, host is ubuntu server with 2.6.24-17, i'm waiting for 2.6.25 which fixes the skb broadcast bug it seems). If I use hwinfo, I can see the virtual disk (over /dev/sdc and the 2nd entry, /dev/sdf). Over fdisk, I get NOTHING.
Here's the dmesg output, hwinfo output, and fdisk output:
On Mon, Jun 02, 2008 at 01:32:15PM -0300, Arturo 'Buanzo' Busleiman wrote:
> On May 30, 10:29 am, Konrad Rzeszutek <kon...@virtualiron.com> wrote: > > If you run iSCSI on each guests you end up with overhead.
> I decided the same. So I'm running iscsi on the host, and via > /dev/disk/by-path/whatever provide the guests with SAN storage.
> Bad thing there, for each virtual disk of the san, I get two /dev > entries, so I'm wondering how to setup the multipath over those two > /dev/disk/by-path entries (one over each controller).
> I also noticed the IPv^ thing, so I ended up specifying the IP by hand, > and using to iscsiadm commands for each discovery / session initiation.
That works. Bit of a hack. Why not just use the IPv4 on both interfaces?
> Also, as you said, link aggregation makes no sense over crossover and > two different interfaces.
Well, you could use load balancing or play with ifaces with your two NICs and take advantage of the two Ethernet cables from your target.
What this means is that you can setup the /dev/sdc to go over one of your NICs, and the /dev/sdf over the other. For that look in the README file and read up about ifaces. This is the poor man fine-grained NIC configuration.
Or you can use load balancing where you bond both interfaces in one, but for that you need a switch.. And the same for link aggregation or link failure.
But as said before, you are doing cross-over so go with ifaces to take advantage of setting up two sessions on both NICs.
> What I'm NOT doing, is LVM. I wanted to go one layer at a time, and > adding LVM was too much for my limited time in here.
> So, currently, I have two remaining issues:
> 1) setup multipath
That is pretty easy. Just install the package and the two block devices (or four if you are using a dual-controller) will be /dev/mapper/<some really long UUID> on which you can use LVM.
> 2) **URGENT**: I've added a second virtual disk and mapped it to my host > (SAN is an MD3000i, host is ubuntu server with 2.6.24-17, i'm waiting > for 2.6.25 which fixes the skb broadcast bug it seems).
Huh? What skb broadcast bug?
> If I use hwinfo, I can see the virtual disk (over /dev/sdc and the 2nd > entry, /dev/sdf). Over fdisk, I get NOTHING.
Keep in mind that depending on what version of multipath you install you might not have the 'rdac' path checker or that the path priority program is called differently. Get the latest one and see what config options you need.
That looks wrong. How many controllers do you have? I wonder if this is related to the other session that didn't login. Have you set the LUN1 disk to have a preferred controller which is the one using IPv6 and the LUN0 to use the controller which is using IPv4?
Can you attach the output of 'iscsiadm -m session -P 3'? And the iscsiadm -m discovery -t st -p 192.168.131.101
> Might the disk still be initializing? The Dell client says it's finished...
Nope. That just means you need to get the latest multipath code to take advantage of your RDAC controller. The other block device isn't activated and if you use multipath it will be accessible (well, kind of . It will be a "passive" disk - meaning if the connection to the active dies the I/O will switch over to the passive disk).
How many controllers are in your MD3000i? If you have two, make sure you're logging in to the interfaces on the controller that is currently the preferred controller (you can tell in MDSM). That dmesg output and not being able to use fdisk against the disk is exactly what I get when I log in to the interfaces on the standby controller.
-----Original Message----- From: open-iscsi@googlegroups.com [mailto:open-iscsi@googlegroups.com] On Behalf Of Arturo 'Buanzo' Busleiman Sent: Monday, June 02, 2008 9:32 AM To: open-iscsi@googlegroups.com Subject: Re: Design Questions
On May 30, 10:29 am, Konrad Rzeszutek <kon...@virtualiron.com> wrote: > If you run iSCSI on each guests you end up with overhead.
I decided the same. So I'm running iscsi on the host, and via /dev/disk/by-path/whatever provide the guests with SAN storage.
Bad thing there, for each virtual disk of the san, I get two /dev entries, so I'm wondering how to setup the multipath over those two /dev/disk/by-path entries (one over each controller).
I also noticed the IPv^ thing, so I ended up specifying the IP by hand, and using to iscsiadm commands for each discovery / session initiation. Also, as you said, link aggregation makes no sense over crossover and two different interfaces.
What I'm NOT doing, is LVM. I wanted to go one layer at a time, and adding LVM was too much for my limited time in here.
So, currently, I have two remaining issues:
1) setup multipath 2) **URGENT**: I've added a second virtual disk and mapped it to my host (SAN is an MD3000i, host is ubuntu server with 2.6.24-17, i'm waiting for 2.6.25 which fixes the skb broadcast bug it seems). If I use hwinfo, I can see the virtual disk (over /dev/sdc and the 2nd entry, /dev/sdf). Over fdisk, I get NOTHING.
Here's the dmesg output, hwinfo output, and fdisk output:
