Gimpster & His Boring Blog

Making Exchange 2007 perform on ESX.

For a couple years we’ve maintained a full lab environment of our production hosted Microsoft Exchange 2007 CCR cluster. I have to be honest, and the Exchange administrator would agree, its never really performed that well. Finally the Exchange admin pretty much got ticked at the performance the other day and while he was out on vacation I thought I’d see what I could do. Its now performing about 100x the speed it ever did, and when he comes back Monday I hope he’s happy. More than anything we’ve just not had the time to really dig into the issue(s).

Needless to say over these couple years we’ve all learned quite a bit about how to eek more performance out of ESX, and in particular Exchange on ESX. I thought I’d share a bunch of the concepts and tidbits in one spot I used to arrive at better performance. The old tricks of throwing RAM and vCPU’s at the problem just didn’t cut it.

Its worth noting that people these days might not choose to use CCR on a virtualized environment (or never did), however I feel these concepts bleed over into stand-alone or maybe even FT/vLockStep implementations going forward.

First and foremost, I invite you to read this article on Exchange 2007’s memory managment strategy:

http://msexchangeteam.com/archive/2008/08/06/449484.aspx

Okay, now that you’ve read that lets continue. Suffice it to say Exchange literally grabs every piece of memory and page it can…. if you let it (which most people do for cache/performance reasons).

As most of you reading know, ESX has quite a few tricks up its sleeve in the memory management department itself and I invite you to read about those concepts in the vSphere/ESX manuals. Obviously the VM’s have access to physical RAM, shares RAM pages when possible, begs/borrows/steals from other VM’s (balloon) and when necessary swaps to disk as a last resort. After careful examination of the performance logs of the Exchange VM’s, it became very obvious ESX was swapping.

Now, we happen to have an entire lab cluster and nobody really cares about performance … well nobody except the persnickity Exchange admin anyway. After doing some research I came to the conclusion I didn’t want the Exchange servers to swap memory … period. I then set a reservation on the VM to the exact same size as the RAM I’d granted. In this case 3GB. This effectivly disables the vswap since the host has no choice but to ante up.  Doing  just this provided an incredible performance boost to the Exchange cluster, but I scratched a little deeper.

Within the VM itself I observed that it was paging quite a bit, and as the above referenced article shows… it always will. So to extract maximum paging performance I decided to create a couple LUN’s and mapped them raw (RDM) to the Exchange servers. I then did some research and came to the conclusion 4kb was the optimal block size for a raw paging volume and if anyone has differing opinions on that PLEASE post them. I then created page files equal to granted memory +20MB. After doing all of that and rebooting, I could tell we’re really cooking with gas now.

Going forward I would like to talk to the Exchange adminstrator about migrating his VMDK based message stores to RDM as well.

So, in closing … hopefully some of these ideas will help you come up with your own Exchange performance issue resolution. I’m sure there are more tricks I need to find, but right now I’m pretty happy with the results.

Till next time…

ESX, Linux LVM, file system expansion and you.

I see quite a few methods being used out there for logicial volume expansion. Everything from what I’m about to post all the way down to folks using cloning tools like Ghost and Clonezilla. Heres how I do it, and it works good for me. Note that I typically do these offline, for no other reason than I’ve biffed enough servers in my life to not be in a rush. If you’re needing 5-9’s uptime, look for another blog. It seems more than anything I have to expand / and so that also can play into this.

Likely a good idea to visit the LVM man pages and/or this site if your LVM is a bit weak: read this and maybe also this.

1. Shutdown the VM.
2. Extend the VMDK you’re working with. You have a couple choices here. You can either extend it from the service console like so: vmkfstools -X 5G /path/to/the/vmdk (would add 5GB to that VMDK) or you can also extend it in the Virtual Infrastructure Client. If you use the service console, look at the options for vmkfstools for the exact syntax for your case.
3. I think it goes without saying you should have backups and/or snapshots of the VM.
4. Boot the LiveCD for your particular flavor of Linux. There are LVM LiveCD’s out there too, and also I bet a Knoppix disc might even work if in a pinch. I happen to run SLES & openSUSE pretty much exclusivly and the LiveCD’s work pretty well.
5. Use fdisk and partition your new found space properly. Make sure you change the type to LVM (8e). If you’re not savvy with fdisk, you should likely stop now.
6. So lets say the new partition you created is /dev/sda5. You now need to create a new physical volume, use the command: pvcreate /dev/sda5 and with any luck it’ll complete.
7. Now you need to extend the volume group to the physical volume you just added. Use the command: vgextend *volumenamehere* /dev/sda5. Use vgdisplay if you don’t know which volume group you need to extend off the top of your head.
8. Finally, its time to extend the logical volume itself. Use this command: lvextend -L 5GB /your/logical/volume/here. Use lvdisplay if you don’t know which LV you want to extend by heart. In this example I added 5GB to the volume.
9. At this point, if everything worked, you can expand the file system. On the systems I administrate we typically use XFS these days. XFS requires the filesystem to be online to grow. So while in the LiveCD you can quickly mount say /dev/system/root to /mnt and then execute xfs_growfs /mnt. Research the proper way to grow whatever file system YOU happen to use.
10. Reboot the VM and make sure all is well.

Till next time…

Why did sysprep get so hard for Server 2008, Vista and Windows 7?

I know I’m way behind the curve on this, but why did sysprep have to get so difficult on the newer server and workstation operating systems? For quick hit deployments, the 20 line .ini file worked just fine. XML, the wave of the future. I’m thinking about writing a quick webpage that cranks out the XML for you without having to crack open WIM’s and install an almost 3GB deployment toolkit.

That’s really all I had to say after a long lapse in posting

Till next time…

Evicting a broken cluster node via command line.

Like I’ve said before, I mostly have this blog for me. I often forget how to do stuff due to getting old.

Ran into a situation where a Microsoft Cluster node ate itself, and I wasn’t able to evict it with the GUI Cluster Administrator. This second proceedure at this link fixed it up: http://support.microsoft.com/kb/282227

If you cannot start the Cluster service, or if you have trouble removing the node, you can manually unconfigure the Cluster service:

1. Run the Cmd.exe program to open a command prompt.
2. At the command prompt, type cluster node nodename /forcecleanup, and then press ENTER.

Quick & dirty PXE boot server using openSuSE.

Lately I’ve been dinking around with Clonezilla and other free disk imaging systems (more on that later), and using CD’s or USB keys can get tiring very fast. Besides Solaris Jumpstart servers, I’d really never gone down this path before. Turns out its quite simple, and you should be able to get it going in 20-30 minutes tops!

I’m going to assume the readers of this have some cursory knowledge here. It should be noted that I added a DNS & Squid server to the mix here only because my server build VLAN is isolated from the world. I wanted to be able to patch them current after the image was deployed. If you don’t have those requirements, don’t install them.

Quick network overview: My PXE boot server VM has one foot in a network with internet access, and the other foot is in an isolated build network. From this isolated network I image, patch and then scan the new machines for vulnerabilities.

1.) Install the operating system. For this example I used openSuSE 11.1 on a pretty low-rent VM. Think 256MB of RAM and 4GB of disk or there abouts.

2.) After you have the operating system installed, add the following packages with yast:

• bind
• yast2-dns-server
• atftp
• syslinux
• dhcp-server
• yast2-dhcp-server
• squid
• yast2-squid

Yast will then complain about needing a bunch of dependencies, so just let it rip.

Okay, so now that you have all of that installed.

3.) Configure the DNS server with the appropriate forwarders (if you want) and all that jazz. However you like just so that the server can resolve. If you want, setup a fake dns zone like serverbuild.local. You could even setup a reverse zone for your bogus privately IP’d build network if you’re feeling really froggy.

4.) Configure & start the DHCP server with appropriate IP range, IP to bind to (you most likely don’t want to have DHCP server on your public side!) and dynamic DNS to the aforementioned configured DNS server, should you so choose. The last three lines are PXE boot/tftp boot specific. Here is my /etc/dhcpd.conf:

option domain-name “serverbuild.local”;
option domain-name-servers 10.0.0.1;
default-lease-time 14400;
ddns-update-style none;
subnet 10.0.0.0 netmask 255.255.255.0 {
range 10.0.0.10 10.0.0.240;
default-lease-time 14400;
max-lease-time 28800;
server-name “10.0.0.1″;
next-server 10.0.0.1;
filename “/tftpboot/pxelinux.0″;
}

5.) Configure & start the TFTP server. Once again I’d restrict this to your private side, unless you really want a TFTP server open to lots of folks (you don’t). Here is my /etc/sysconfig/attftpd:

## Path:    Network/FTP/Atftpd
## Description: ATFTP Configuration
## Type:    string
## Default: “–daemon ”
#
# atftpd options
#
ATFTPD_OPTIONS=”–logfile /var/log/atftpd.log –daemon –user tftp -v”

## Type:    yesno
## Default: no
#
# Use inetd instead of daemon
#
ATFTPD_USE_INETD=”no”

## Type:    string
## Default: “/tftpboot”
#
#  TFTP directory must be a world readable/writable directory.
#  By default /tftpboot is assumed.
#
ATFTPD_DIRECTORY=”/tftpboot”

## Type:    string
## Default: “”
#
#  Whitespace seperated list of IP addresses which ATFTPD binds to.
#  One instance of the service is started on each IP address.
#  By default atftpd will listen on all available IP addresses/interfaces.
#
ATFTPD_BIND_ADDRESSES=”10.0.0.1″

6.) Hopefully I’m coherent enough on the following steps:

If it doesn’t already exsist, make a /tftpboot directory and copy /usr/share/syslinux/pxelinux.0 to /tftpboot. While you’re at it also make a directory called /tftpboot/pxelinux.cfg.

7.) Create a file with the following contents named /tftpboot/pxelinux.cfg/default:

default harddisk

# hard disk
label harddisk
localboot 0×80

# clonezilla live
label clonezilla
kernel vmlinuz1
append initrd=initrd1.img boot=live union=aufs noswap noprompt ip=frommedia fetch=tftp://SERVERIPHERE/filesystem.squashfs

implicit 1
gfxboot bootlogo
display message
prompt 1
timeout 600

Change the SERVERIPHERE to represent your TFTP servers private IP.

8.) Obtain the latest Clonezilla live cd .zip file, or one you’ve created custom (more on that later!). Unzip the .zip file in a location of your chosing and then copy live/vmlinuz1, live/initrd1.img and filesystem.squashfs to /tftpboot.

In case you’re not in the know, vmlinuz1 = the kernel, initrd1.img = initial RAM disk image and filesystem.squashfs = the file system with all of the fun clonezilla stuff in it. I’m over simplifying things here, but whatever.

For more details, see this link: http://www.clonezilla.org/clonezilla-live/livepxe.php

9.) Create the message file you’ll see when your machine PXE boots named /tftpboot/message. This is what mine contains:

Clonezilla PXE Boot

To start the imaging system enter ‘clonezilla’ and press enter.

Available boot options:

harddisk – Boot from Harddisk (this is default)
clonezilla – CloneZille Live CD

Assuming you have all of the services started, you should now see this menu on the server(s) you wish to image. Type clonezilla and in theory the Clonezilla build environment should begin to load. This all works very nicely on my IBM blade servers.

While in this example I focused on Clonezilla, you could replace (or add) any linux kernel & initrd (plus any other associated files) your little heart desires in the /tftpboot/pxelinux.cfg/default. Like for example a RedHat server build environment or whatever you might need. I assume you could easily shove in a BartPE/WinPE image too if you wanted.

I’ll run through this again later in the week to make sure I didn’t miss something huge.

I didn’t really get into configuration of the Squid proxy I guess, but there are plenty of HOW-TO’s out there for that!

Till next time…

Restoring default config on iPhone MobileTerminal.

I see this issue/question posted quite a bit, and after helping several people on various forums… I thought this might help others too.

If you’ve used MobileTerminal on your jailbroken iPhone, then you should be aware of the customization possible. More specifically: customization relating to the command shortcut menu. While customizing this and other menus, it’s very easy to accidentally wipe out an entire submenu! The best example I can think of would be picking say the “ls” submenu, and then accidentally hitting off on the submenu instead of the right arrow to modify it.

Another common menu to wipe out is the “misc” menu. Never fear though, you can still access the configuration menu by taping the AT&T or iPod logo in the upper left corner.

See the below screen shot if I was confusing:

At this time, MobileTerminal doesn’t have a graphical method to restore defaults. The quickest way out of this mess is to open MobileTerminal and issue the following command:

rm -rf /private/var/mobile/Library/Preferences/com.googlecode.mobileterminal.* The case is sensitive, so type that in exactly as posted.

Restart MobileTerminal and you should be good to go. Consequently, that’s what you’d also want to backup if you were about to blow away your iPhone and wanted to keep your tricked out MobileTerminal preferences.

Till next time…

Getting show desktop back into your quick launch.

I’m not sure why, but for whatever reason the Show Desktop shortcut disappeared from my quick launch menu. You’d think this would be really easy to get back, but either I’m stupid or it isn’t that easy. I’m sure there’s some more elegant way to do this, but I never claimed to be an expert. Whatever the case, not having this shortcut really bugs me … too lazy for keyboard shortcuts

Anyway, here’s how I got mine back.

Start -> Run -> notepad

Paste in the following and save it as Desktop.scf:

[Shell]
Command=2
IconFile=explorer.exe,3
[Taskbar]
Command=ToggleDesktop

Copy / paste this file into: C:\Documents and Settings\your_user_here\Application Data\Microsoft\Internet Explorer\Quick Launch

Then right click on the task bar and disable/enable Quick Launch to force a re-read.

Till next time…

Part 2: Where’d my IO go?

Shortly after upgrading my Hackintosh to 10.5.6 it occured to me that the machine was NOTABLY slower. Digging into it a little bit more I discovered it was all IO wait issues (read: poor disk performance). We’re talking like 5 minute boots and fun stuff like that.

I opened up System Profiler, and to my surprise there were no longer any SATA devices (T43 is a SATA based machine, at least mine is). The hard drive was now nested in the ATA section with the DVD-ROM/CD-RW combo drive. At this point I suspected driver schnanigans so I then dumped dmesg to a text file and put it on a usb keychain for some comparisons.

Once again I imaged the machine and then restored the old 10.5.3 image and compared dmesg. The old 10.5.3 (which FLEW compared to my 10.5.6 image) was using AppleIntelPIIXATA.kext and the 10.5.6 image had switched to AppleVIAATA.kext. As I said in part 1, I had to select VIA-SB to even get OSX to install on this machine… so I guess this makes sense. I haven’t had time to boot the install DVD again to see if PIIXATA is an option to choose.

ANYWAY, I found this thread on InsanelyMac which includes the PIIXATA drivers for various OSX builds: AppleIntelPIIXATA kext fully working for all ICHx Mobo (All Sata Channels working), Fully working kext (Leopard & Tiger version 1.1)

I don’t think English is his first language, so let me try to clarify:

• Use OSX86Tools to install the AppleIntelPIIXATA.kext, don’t close the app yet.
• Open terminal and change directory to: /System/Library/Extensions
• mv AppleVIAATA.kext /var/tmp (in case you need it back!).
• Back in OSX86Tools check Set Extensions permissions & Clear Extensions Cache and then Run Selected Items.
• Reboot machine, and you should notice GREATLY improved performance.

If it doesn’t boot again, well don’t blame me

Till next time…

Part 1: Getting OSX 10.5.6 on my IBM T43 laptop

Over the next few posts I’m going to chronicle my struggles to get OSX 10.5.6 on my test laptop, this is just the first installment.

As I might have posted in the past, I do a bit of testing/development for OSX on a Hackintosh, more specifically using an IBM T43. I’m somewhat ashamed to say that, being that I’m an iPhone/iPod owner (and generally a fan of Apple products). I would love to own a real Apple branded machine, but they’re just not within my budget. Even used mini’s still seem to fetch near MSRP prices on eBay. So please, I really don’t need any lectures.

I started this journey with Kalyway 10.5.1 and over time used their comboupdates to get it to 10.5.3. I’ve been very content for a long time, but lately I needed to do some testing that required 10.5.6.

I imaged the laptop and unsuccessfully tried about 50 different methods to get it to 10.5.4, .5 and .6. Nothing really worked out for me 100%, so I decided to turn to the newer iDeneb releases.

First I acquired the iDeneb 1.4 (10.5.6) DVD ISO, and attempted to install. The DVD boots, installs about 3/4 the way and then dies with this error: pkgExtractor[162]: BomFileError 2: No such file or directory – ./usr/X11/share/doc/renderproto/renderproto.txt. I dicked with this thing for days and on about 7 different machines (not T43’s) and recieved the same results. Either I’m doing something TOTALLY wrong, or this ISO is bunk. I found a few other people who were having the same issues on the InsanelyMac forum, but no fix was ever posted.

Then I acquired the iDeneb 1.3 (10.5.5) DVD ISO, and likewise attempted to install … success! The winning combination to choose at install is:

• X11
• Patches 10.5.5 Ready -> Audio -> AppleAzailiaAudio
• Patches 10.5.5 Ready -> Chipset -> VIA-SB
• Patches 10.5.5 Ready -> Kernel -> Kernel 9.4.0 StageXNU
• Patches 10.5.5 Ready -> Network -> AppleBC5751 (more on this later)

If you’ve never installed OSX on this laptop before, make sure you disable SpeedStep in the BIOS or you’ll be hating life.

At this point you more/less have a working 10.5.5 machine, which is a step in the right direction. Still isn’t 10.5.6 though, so let’s continue on. You’ll notice your video isn’t quite right yet, and maybe some hardware you have isn’t working right either. My suggestion: get it to 10.5.6 and THEN worry about that stuff.

Lucky for us, iDeneb released a 10.5.6 upgrade kit. It can be found here: http://ideneb.net/en/homepage/63-ideneb-upgrade-kit-1056-rev1

Follow the instructions with it and voila, you’ll be at 10.5.6.

Now, to get all of my hardware devices hmmmmmmmmmmmm……….

Till next time…

Manually reactivate LVM’s in Linux

I’m not sure what the deal is, but often when you boot Linux up single user you only get the root volume group activated. This could be totally normal behavior, and no big deal. When you have /usr /var and other portions of the file system broken up … it becomes a big deal.

After looking around the lvm man page awhile, this worked for me.

Before:

# lvm lvscan
Failed to find sysfs mont point
inactive ‘/dev/var/var [7.99 GB] inherit
ACTIVE ‘/dev/system/root [7.80 GB] inherit
ACTIVE ‘/dev/system/swap [2.00 GB] inherit

Execute this:

# lvm vgsan
# lvm vgchange -ay

After:

# lvm lvscan
Failed to find sysfs mont point
ACTIVE ‘/dev/var/var [7.99 GB] inherit
ACTIVE ‘/dev/system/root [7.80 GB] inherit
ACTIVE ‘/dev/system/swap [2.00 GB] inherit

You may now mount volumes as normal for maintenance.