• To quote a recent Oracle publication, “Oracle plans to engineer and deliver an integrated system – applications to disk – where all the pieces fit and work together, so customers do not have to do it themselves.” Sun is already shipping Infiniband switches and blades with InfiniBand on the motherboard. They have also mentioned IB is on the roadmap for the Sun 7000. Andy Bechtolsheim mentioned it at the Sun product announcement on April 14th. What about an integrated Oracle appliance running on Nehalem blades, Solaris x64, Sun 7000 storage, and using Infiniband switches. It should not be a major technology leap to put it all together. What would this mean for the Oracle/HP appliance?
• Sun SPARC processors are at an Oracle pricing disadvantage to IBM Power processors in the current Oracle pricing model. Oracle has never been afraid to use pricing to move the market in their direction. Watch for them to use their pricing model to encourage customer to buy Sun servers.
• Solaris x64 has been intentionally neglected by Oracle. Oracle delivers patches on Solaris SPARC and Linux immediately. Then, they have historically waited up to 6 months to release that same patch for Solaris x64. In the past, this has helped Oracle push their Linux agenda in the marketplace. Given the ease of porting the Oracle patches to Solaris x64, there is no logical technical reason for this lag. Watch for Solaris x64 to become a first class citizen in the Oracle OS support matrix now that growth of Solaris x64 means growth for Oracle.
• Storage – Sun is not generally thought of as a storage company. However, an Oracle executive recently announced in a Sun all hands meeting that Oracle was buying Sun for Solaris, Java, and storage. Look for the Sun 7000 to pick up some new features to integrate tighter with Oracle. Perhaps Analytics that give visibility across both Oracle, Solaris, and the storage? Oracle performance could benefit from the large cache footprint and even larger flash cache. The Sun 7000 is trying to reduce hardware costs and in the past Oracle has loved reducing hardware costs in order to leave more budget available for software.
• Servers – I would expect business as usual at Sun on the server front. Oracle has no play in the hardware space today and Sun makes solid products. Sun’s server business generates positive cash flow every quarter. This is part of why they bought Sun, right? According to a recent release from Oracle, “Oracle plans to grow the Sun hardware business after the closing, protecting Sun customers’ investments and ensuring the long-term viability of Sun products.”
• MySQL – Many users adopt MySQL to escape Oracle pricing. The largest market is in the Web 2.0 space, but they play in other parts of the market as well. There is no way to kill MySQL because it is open source. If Oracle/Sun stopped supporting it today, there would be a well funded startup tomorrow stepping in to take their place. Much of the MySQL technical talent has already left Sun. In fact, some major MySQL users are turning to the community and other sources for their MySQL support and patches. Oracle already owns TimesTen, Berkeley DB, and InnoDB, so MySQL could just become another DB on the list. If Oracle really wants to put the pressure on Microsoft SQL, then they will encourage the use of MySQL and provide an easy migration path to Oracle DB.
• The Wall Street Journal reported that the IBM deal fell through when the Sun board split into two factions. The one in favor of the IBM deal was led by Jonathan Schwartz and Scott McNealy headed the group opposing the acquisition. When the CEO takes that kind of stand and loses, one must wonder what this foretells of his future. I think there is a high likelihood we will see Jonathan move on to a new opportunity in the next few months.
• Identity Management – Oracle and Sun have overlapping products here. When both products are ‘good enough’ I tend to lean towards the buyer. That said, Sun has their product somewhat integrated into Solaris, so I do not think it will go away. This one may be too close to call. There may also be some regulatory scrutiny on this as the combined market share will be significant.
• Sun IBIS – Sun has been trying to centralize to a single ERP system over that last few years. This is a project that should never have impacted Sun’s customers, partners, or suppliers. Unfortunately, there have been a few hiccups and many delays along the way and it has effected many organizations outside of Sun. Look for Oracle to bring the resources to the table to clean this up.

I think we would have seen some serious regulatory scrutiny of the IBM acquisition of Sun. On the surface, it looks like there are very few issues with Oracle and Sun. The only two issues I see are Identity Management and mySQL. I don’t think either one should be a major issue, but the government may. I don’t think Oracle will have a problem finding a workable solution for either potential objection.

A couple thoughts that are a little further out of the box:

• An Oracle appliance would give Oracle a private cloud play in the datacenter. Oracle has a large cloud presence today with Oracle On Demand. They could position themselves as the enterprise application private and public cloud provider. Cloud computing is the natural evolution of outsourcing. Oracle could compete with IBM Global Services on the high end and Amazon on the low end. Who better to outsource your enterprise applications to than the company that wrote them?
• DTrace could be integrated into Oracle to provide a whole new level of observablity. What if DTrace had a GUI that was able to observe systems across the database, server, and storage?
• Oracle is based on transactions and ensuring transactional integrity is maintained all the way from the Oracle DB to the hard drive. Sun’s zfs is a transaction engine for storage that is most commonly used to present a filesystem. The zfs transactional model is completely exposed through their API. What would hapen to Oracle performance if they plugged into the zfs transactional engine and took advantage of the zfs integration with SSD/Flash?
• Oracle could take OpenOffice to market and try to compete with Microsoft for the desktop. I am not suggesting that OpenOffice is perfect or as full featured as Microsoft Office, but Oracle has a history of monetizing software very effectively.
• Oracle and Sun have both been working on Xen. I assume they are both working on tools to manage virtual environments. Do they try to enter the virtualization space? The hypervisor is well on the way to commoditization (read: free), but what about a virtualization appliance. Sun has the hardware and Oracle does a pretty good job on the software side.
• Sun is a major proponent of Infiniband. One of the potential limiting factors in Oracle RAC environments is the latency of the cluster interconnect. What impact would Oracle’s endrsement of Infiniband have in the marketplace? How would this impact Cisco’s Data Center Ethernet (DCE) plans?

What are a couple of the potential challenges in this acquisition?

• A few analysts have suggested that Oracle will sell off the Sun hardware business. I can not see it. Why would they do that? As mentioned above, Oracle states in the Oracle/Sun FAQ, “Oracle plans to grow the Sun hardware business after the closing, protecting Sun customers’ investments and ensuring the long-term viability of Sun products.”
• Oracle has never had hardware in their portfolio. How does this affect their relationships with IBM, HP, Dell, EMC, and NetApp?
• Pillar is a small, but very well funded, storage company. Tako Ventures is the largest investor in Pillar and has a seat on the board. Lawrence Investments is the owner or Tako Ventures and Lawrence “Larry” Ellison provides the capital behind Lawrence Investments. So, Larry will own two storage companies when the Sun deal closes.
• Sun and Oracle have very different cultures at the field level. How will that play out? Given Sun’s latest reorganization, I think the sales organization could be plugged into a new company quite easily. It almost appears like it was designed to be portable. Perhaps that was a design criteria?

How does it affect the rest of the industry?

Once the IBM news broke, it was generally expected that Sun would be acquired. Even if the IBM deal fell through, someone else would buy Sun. I expected it to be one of the major systems vendors such as HP or Dell. If I let my mind wander a bit, I could convince myself that Cisco or EMC might take the opportunity to enter a new space. I did not expect Oracle to jump in and make the deal. I don’t think the rest of the industry expected it either. Any company who makes a living selling hardware for enterprise applications in the datacenter is unhappy about this acquisition. Oracle is the largest vendor of enterprise application software. When they sold no hardware, their support matrix and pricing model would drive hardware buying decisions. Now that they will have their own hardware portfolio, I expect them to push customers towards Solaris, Sun storage, and Sun SPARC and x64 servers.

The current state of the economy has driven down stock prices. This is a buying opportunity for companies with a war chest. Cisco took out a $4B bond a couple months back, which suggests they are shopping. IBM, EMC, Dell, HP, NetApp, and Cisco and have money to invest. Sun is the first major transaction of the year, but I do not think it will be the last.

Full disclosure: My company is a partner of both Sun and Oracle.

Disk drive capacity is growing exponentially, but disk performance is increasing at a much slower rate. In many cases, when helping customers size for their workload, performance drives the spindle count and not capacity. It is easy to meet the capacity needs with large drives, but will they meet the performance requirement? That is the problem. Often performance is what dictates the spindle count. It is no longer sufficient to size a storage device based solely on capacity requirements. This is a general challenge that must be taken into account when sizing a storage array.

So how is the growing disparity between size and performance effected by deduplication? Deduplication can make the performance issue worse by reducing the number of spindles even further. If the bottleneck in the storage device is the spindles, then using deduplication to pack more data onto those spindles is only going to exacerbate the situation.

Let’s take a closer look at sizing storage for a backup to disk workload. Delivering on the highly sequential read and write requirements of disk to disk backups is much easier than serving a more random workload. Disk drives do a great job with sequential reads and writes. This makes backup to disk all about sizing for capacity. When deduplication is added into the mix, the disk drives should still meet the performance requirement as long as the deduplication technology being used does not turn sequential IO into random IO. This is why it is important to understand how a specific deduplication implementation works.

The reality is that nearly every other IT workload is more random than backup to disk. If deduplication was used to pack more data onto the same number of spindles for a highly random workload the spindles would likely not meet the performance requirements. Does that mean deduplication is a point solution for highly sequential workloads? I do not believe so. I am working on an entry covering the potential performance benefits of deduplication in a more random environment.

In moving to NUMA Intel has joined other CPU designs:

• Sun’s UltraSPARC uses a shared parallel bus called the Sun Fireplane Interconnect Bus combined with a Crossbar switch architecture for even larger multi-tiered NUMA fabrics
• AMD’s Opteron uses a non-shared serial transport called the HyperTransport Bus, which limits its scalability to create even larger multi-tiered NUMA fabrics based on the limited number of Hypertransport connections available
• IBM’s P6 uses a combination of non-shared serial Inter-Chip transport (for connecting 4 CPU’s together into a fast NUMA node) as well as serial Inter-node transport (for connecting 4 CPU NUMA nodes together into an even larger multi-tiered NUMA fabrics)

NUMA as a trend should continue unabated. NUMA and virtualization complement each other extremely well, especially when you consider that a virtualized machine will typically run within its own NUMA node unaffected and unimpeded by virtual machines running within other NUMA nodes on the box.

NUMA and HPC is another area that will only see gains. Using larger 8+ CPU boxes in an HPC design will yield higher marks than many smaller 2 CPU boxes. Intel’s switch to NUMA will allow for Intel boxes that can contain more than 4 CPU’s without the need for special chipsets that separate the FSB and provide a cache coherency mechanism between the FSB’s.

NUMA architectures put more pressure on operating systems. An operating system needs to understand the performance “distance” of various components of the system to it can properly schedule CPUs and allocate memory to optimize performance. The higher the number of threads and interconnected components the more important this is.

Thanks to Ed Hamilton, Principal Solutions Architect at Corporate Technologies, Inc., for much of the content in this blog posting.

The Nehalem announcement was different, in that Intel was solving the bus limits that have impacted overall system throughput. The announcement was exciting but being able to buy systems containing those CPUs took a long time to reach fruition. Pre-release performance tests of these new Xeon 5500 CPUs revealed that the announcement was more than marketing. They show that Nehalem provides breakthrough increases in many aspects of system throughput. In many areas, the 5500 is twice as fast as its predecessor 5400. Unfortunately the 5500 is not a drop-in replacement for other Xeon CPUs. It requires a new chip set and socket.

The enhanced feature set and performance are going to make it worth the effort to roll out new systems and motherboards. The first shipping CPUs are quad-core. Memory I/O is now up to 3X faster. The CPUs are more “green” in that they detect how much work is being requested and power-down unused components. They can also increase performance in a given core if other cores are not needed. Especially exciting for virtualization users is that the hardware provides more virtualization assistance in the form of Intel VT-c. I/O and paging in virtual environments should be faster as virtual machines no longer need the host OS to do page table lookups for them.

As of today, Intel is shipping those CPUs. Specifically they are shipping 1 and 2 socket EP versions. Apple has previously announced Nehalem-based workstations, but today many vendors announced servers. The 4-socket EX servers are expected later this year and will lead to another round of price / performance / feature-set / density leading servers.

eWeek.com has a good summary of the CPU architecture and its benefits. The Intel web site includes whitepapers and videos about the Nehalem architecture. ars technica provides a good CPU overview as well as a detailed review of the single-socket Nehalem. Real World Technologies wrote an article describing all the gory technical details of Nehalem.

Backup to disk is the number one application for deduplication today. My presentation covered the most common approaches vendors use to leverage deduplication in a backup environment. This includes client side, backup server based, inline processing, and post-processing. The slides are available here.

This topic was originally developed as part of a CTI Strategy Services consulting engagement. The customer credited us with saving them 6 months of effort.

Will more cache improve the performance of my storage array? This is what the vendors so often tell us, but they have no objective information to explain why it is going to help. Depending on the workload, increasing the cache may have little or no effect on performance.

There are two ways to know whether your environment will benefit from additional cache. The first is to understand every nuance of your application. Most storage managers I speak with classify this as impractical at best and impossible at worst. Even if you have an application with a very well understood workload, most storage devices are not hosting a single application. Instead, they are the hosting many different applications. It is even more complex to understand how this combined workload will be effected by adding cache.

The second way to measure cache benefit is to put the cache in and see what happens. This is the most common approach I see in the field. When performance becomes unacceptable, the options of adding additional disk and/or cache are weighed and a purchase is made. (I will save the topic of adding spindles to increase performance for a future post.) Both of these options force a purchase to be made with no guarantee it will solve the problem.

NetApp has introduced a tool to provide a 3rd option: Predictive Cache Statistics. It provides the objective data needed to rationalize a hardware purchase. Predictive Cache Statistics (PCS) is available in systems running 7.3+ and having at least 2GB of memory. When it is enabled, PCS reports what the cache hit ratio would be if the system had 2x (ec0), 4x (ec1), and 8x (ec2) the current cache footprint. (ec0, ec1, and ec2 are the names of the extended caches when the stats are presented by the NetApp system.)

Now, let’s drill down into exactly how predictive cache statistics work…

In most conditions there is no significant impact to system performance. I monitored the change in latency on my test system with PCS enabled and disabled and there was not a measurable difference. The storage controller was running at about 25% CPU utilization at the time with a 40% cache hit rate. NetApp warns in their docs that performance can be effected when the storage controller is at 80% CPU utilization or higher. It is understandable given the amount of information the array has to track in order to provide the cache statistics. This simply means some thought needs to be put into when it is enabled and how long it is run for in production.

Here are the steps required to gather the information:

1) Enable Predictive Cache Statistics (PCS)

options flexscale.enable pcs

2) It is important to allow the workload to run until the virtual caches have time to warm up. In a system with a large amount of cache, this can be hours or even days. Monitor array performance while the storage workload runs. If latency increases to unacceptable levels, you can disable PCS.

options flexscale.enable off

3) The NetApp perfstat tool can be used to capture and analyze the data that is gathered. I prefer instant gratification, so for this example, I will use real time stats command.

stats show –p flexscale-pcs

The way the results are reported can be a little confusing the first time you look at it. The ec0, ec1, and ec2 ‘virtual caches’ are relative to the base cache in the system being tested (2x, 4x, and 8x). If the test system has 16GB of primary cache, ec0 will represent 32GB of ‘virtual cache’ (2x 16GB). ec1 brings the ‘virtual cache’ to a total of 4x base cache or an additional 32GB beyond ec0. ec2 brings the total to 8x base cache or an additional 64GB beyond ec0 + ec1. The statistics on each line represent the values for that specific cache segment. Hopefully that explanation clears up more confusion than it introduces.

Here are a couple examples. This testing was completed on a NetApp FAS3170. The 3170 platform has 16GB of cache standard. So, in these examples, ec0 is 32GB, ec1 is 32GB, and ec2 is 64GB.

Example 1: 8GB working set, 4KB IO, and 100% random reads

fas3170-a> sysstat -x 5
 CPU   NFS  CIFS  HTTP   Total    Net  kB/s   Disk kB/s     Tape kB/s Cache Cache  CP   CP Disk    FCP iSCSI   FCP  kB/s iSCSI  kB/s
                                  in    out   read  write  read write   age   hit time  ty util                 in   out    in   out
 39% 39137     0     0   39137  7102 165539    206    370     0     0   >60  100%   3%  T    2%      0     0     0     0     0     0
 39% 39882     0     0   39882  7236 168677    136      6     0     0   >60  100%   0%  -    1%      0     0     0     0     0     0
 39% 39098     0     0   39098  7094 165338    186    285     0     0   >60  100%   3%  T    2%      0     0     0     0     0     0

fas3170-a> stats show -p flexscale-pcs
Instance    Blocks Usage   Hit  Miss Hit Evict Invalidate Insert
                       %    /s    /s   %    /s         /s     /s
     ec0   8388608     0     0     0   0     0          0      0
     ec1   8388608     0     0     0   0     0          0      0
     ec2  16777216     0     0     0   0     0          0      0
---
     ec0   8388608     0     0     0   0     0          0      0
     ec1   8388608     0     0     0   0     0          0      0
     ec2  16777216     0     0     0   0     0          0      0
---
     ec0   8388608     0     0     0   0     0          0      0
     ec1   8388608     0     0     0   0     0          0      0
     ec2  16777216     0     0     0   0     0          0      0

The sysstat shows a cache hit rate of 100%. This is exactly what we would expect for an 8GB dataset on a system with 16GB of cache. The stats command shows that PCS is currently reporting no activity. Again, this is exactly what we should expect with a working set that fits completely in main cache.

Example 2: 30GB working set, 4KB IO, and 100% random reads

fas3170-a> sysstat -x 5
 CPU   NFS  CIFS  HTTP   Total    Net kB/s   Disk kB/s     Tape kB/s Cache Cache  CP   CP Disk    FCP iSCSI   FCP  kB/s iSCSI  kB/s
                                  in   out   read  write  read write   age   hit time  ty util                 in   out    in   out
 27% 11607     0     0   11607  2173 49352  27850      6     0     0     3   41%   0%  -   99%      0     0     0     0     0     0
 27% 11642     0     0   11642  2180 49518  28097    279     0     0     3   41%  21%  T   99%      0     0     0     0     0     0
 26% 11413     0     0   11413  2138 48511  27773     11     0     0     3   41%   0%  -   99%      0     0     0     0     0     0

fas3170-a> stats show -p flexscale-pcs
Instance    Blocks Usage   Hit  Miss Hit Evict Invalidate Insert
                       %    /s    /s   %    /s         /s     /s
     ec0   8388608     1    38  8560   0     0          0  14811
     ec1   8388608     0     0  8560   0     0          0      0
     ec2  16777216     0     0  8560   0     0          0      0
---
     ec0   8388608     1    65  6985   0     0          0      0
     ec1   8388608     0     0  6985   0     0          0      0
     ec2  16777216     0     0  6985   0     0          0      0
---
     ec0   8388608     1   100  6922   1     0          0  11899
     ec1   8388608     0     0  6922   0     0          0      0
     ec2  16777216     0     0  6922   0     0          0      0

This data was gathered after the 30GB workload had been running for a few minutes, but just after I enabled predictive cache statistics. The PCS data shows that there are very few hits, but there are a significant number of inserts. This is what we should expect when PCS is first enabled. The sysstat output shows a cache hit rate of 41%.

fas3170-a> sysstat -x 5
 CPU   NFS  CIFS  HTTP   Total    Net kB/s   Disk kB/s     Tape kB/s Cache Cache  CP   CP Disk    FCP iSCSI   FCP  kB/s iSCSI  kB/s
                                  in   out   read  write  read write   age   hit time  ty util                 in   out    in   out
 27% 11238     0     0   11238  2105 47784  27862    286     0     0     4   40%  18%  T   99%      0     0     0     0     0     0
 26% 11371     0     0   11371  2130 48349  27934     11     0     0     4   40%   0%  -   99%      0     0     0     0     0     0
 27% 11184     0     0   11184  2096 47554  27938    275     0     0     4   40%  33%  T   99%      0     0     0     0     0     0

fas3170-a> stats show -p flexscale-pcs
Instance    Blocks Usage   Hit  Miss Hit Evict Invalidate Insert
                       %    /s    /s   %    /s         /s     /s
     ec0   8388608    87  6536   456  93   933          0    934
     ec1   8388608     6   453     3  99     0        934    933
     ec2  16777216     0     0     3   0     0          0      0
---
     ec0   8388608    87  6512   435  93     0          0      0
     ec1   8388608     6   435     0 100     0          0      0
     ec2  16777216     0     0     0   0     0          0      0
---
     ec0   8388608    87  6472   450  93   963          0    964
     ec1   8388608     6   445     5  98     0        964    963
     ec2  16777216     0     0     5   0     0          0      0

Now that the ec0 virtual cache has warmed up, the potential value of additional cache becomes more apparent. The hit rate has gone up to 93% and it is servicing over 6500 operations per second. With 32GB of additional cache, 6500+ disk reads would be alleviated and the latency would be dramatically reduced. These cache hits are virtual, so currently those ‘hits’ are still causing disk reads. Clearly, the additional cache will provide a major performance boost, but unfortunately, it is impossible to determine exactly how it will effect overall system performance. The current bottleneck, reads from disk, would be alleviated, but that simply means we will find the next one.

Additional cache can be added to most NetApp systems in the form of a Performance Accelerator Module (PAM). The PAM is a PCI Express card with 16GB of DRAM on it. It plugs directly into one of the PCI Express slots in the filer. I suspect there a slight increase in latency when accessing data in the PAM over the main system cache. Although, this increase is likely so small that it will not be noticed on the client side as it is a very small portion of the total transaction time from the client perspective. Unfortunately, I do not have first hand performance data that I can share as I have not been able to get access to a PAM for complete lab testing.

It is important to note that a system with 16GB of primary cache and 32GB of PAM cache is not the same as a system with 48GB of primary cache. The PAM cache is populated as items are evicted from primary cache. If there is a hit in the PAM, that block is copied back into primary cache. This type of cache is commonly referred to as a victim cache or an L2 cache. If the goal is to serve a working set without ever going to disk, then that working set needs to fit into extended cache, not the the primary cache plus extended cache.

Predictive cache statistics are a great feature. It gives us the power to answer a question we could only guess at in the past. However, like most end users, I always want more. There are a couple things that I would love to see in the future. First, the PAM cards are 16GB in size. It would be great if the extended cache segments reported by PCS could be in 16GB increments. That would make it even easier to determine the value of each card I add. It would also remove all the confusion around how big ec0, ec1, and ec2 are. The ability to reset the PCS counters back to zero would also be helpful. When testing different workloads, this would allow the stats to be associated with each individual workload.

It is worth noting that this was not a performance test and the data above should be treated as such. Nothing was done to either the client or the filer to optimize NFS performance. In an attempt to prevent these numbers from being used to judge system performance, I am intentionally omiting the details of how the disk was configured.

This increase in replication deployments has created a new challenge for IT. The most common replication solutions move the data over the IP network. That data puts a significant load on the IP network infrastructure. The LAN infrastructure is almost always up to the task, but the WAN is often not able to handle this new burden. While the prices of network infrastructure have come down over the years, big pipes are still an expensive monthly outlay. So, how do we get that data offsite without driving up those WAN costs? WAN optimization technology provides a potential solution.

Not every workload or protocol can benefit from today’s WAN optimization technology, but replication is one that usually gets a big boost. I gathered some data from a client who is using NetApp SnapMirror to replicate to a remote datacenter and deployed WAN optimization to prevent a major WAN upgrade.

The NetApp filer is serving iSCSI, Fibre Channel, and CIFS. The clients are primarily Windows and they run Exchange and MS SQL along with some home grown applications. All of their data is stored on the NetApp storage.

The chart below shows the impact the WAN optimization device had. For the purposes of this discussion, think of the device as having one unoptimized LAN port and an optimized WAN port. The LAN traffic is represented by the red and the WAN by the blue. With no optimization, the traffic would be the same on both sides. The chart shows a dramatic reduction on the amount of data being pushed over the WAN.

Network Throughput

This data was gathered over a 2 week period. The total data reduction over the WAN was 83% over the data in the chart and there was a peak of 93% for one window. Again, this is not what every environment will see, so test before you deploy. In this case, the system paid for itself in less than 12 months with the savings in WAN costs. That is the kind of ROI that works for almost anyone.

I am intentionally not addressing what WAN optimization technology was used in this solution. Last time we tested these devices in our lab, we brought in a half dozen and they all had their pros and cons. That is another topic for another post.

Solaris Ambassador Jeff Victor will be presenting the latest features in Solaris 10 10/08. This list will include ‘ZFS Boot’ – the ability to install Solaris using only ZFS file systems.

Here is a link to the detailed agenda and registration information on the opensolaris.org website. I hope to see you there.

That said, I am going to talk about an issue that some of the storage bloggers seem to be caught up in at the moment. The issue of ‘emulated FC’ vs ‘real FC.’ Let me start off by sharing a few recent posts from other blogs:

Chuck Hollis at EMC writes about the EMC/Dell relationship and takes the opportunity to compare EMC to NetApp. In this case, he is comparing the EMC NX4 to the NetApp FAS2020. The comment in the post that certainly aggravated NetApp is that EMC does “real deal FC that isn’t emulated.” The obvious implications being that EMC FC is not emulated, NetApp FC is emulated, and FC emulation is bad. (This is not a new debate between EMC and NetApp. Look back through the blogs at both companies and you will find plenty of back and forth on the topic.

Kostadis Russos at NetApp has a post explaining why he, not surprisingly, completely disagrees with Chuck.

Stephen Foskett, a storage consultant, posts what I think is an excellent overview of the issues. He cuts through the marketing spin and asks the right questions. His coverage of the topic is so complete, I almost decided not to write about the topic. I will try not to retrace all the issues he covered. I will hit a couple of his high level points in case you have not had a chance to read his post (I highly recommend it though, it is very good.) In summary:

• All enterprise storage arrays “emulate” Fibre Channel drives to one extent or another
• NetApp is emulating Fibre Channel drives
• All modern storage arrays emulate SCSI drives
• Using the wrong tool for the job will always lead to trouble
• Which is more important to you, integration, performance, or features?

So, why am I writing about it? I am writing about it because Chuck posted a very good blog entry about benchmarking a few days later that, to me, contradicts the importance he gave to ‘real FC’ on 12/9. I have never meet Chuck or Stephen, but they both seem to be very technically adept from their postings.

Without trying to put words in his mouth (text on his blog?), the overall theme of Chuck’s post is to make sure you use meaningful tests if you want meaningful results from a storage product benchmark. He is absolutely correct. I could not agree more. How many times have we seen benchmarks performed that were completely irrelevant to the workload the array would see in production?

My question is, if the end result of performance testing with real world applications produces acceptable results, then who cares what is ‘real’ and what is ‘emulated’? The average driver does not worry about how the computer in her car is controlling the variable valve timing. She worries about whether it reliably gets her to work on time.

VMware is selling plenty of virtualization technology that presents devices that are not ‘real.’ I know it is not storage, but why is that any different? Less and less is ‘real’ in storage these days. It is impossible to continue to drive innovation in storage array technology if we are bound by the old ideas of how we configure and manage our storage. With the introduction of technologies that leverage thin provisioning, dependent pointer based copies, compression, and deduplication we need to rethink concepts as fundamental as RAID groups, block placement, and LUN configuration. Or in my opinion, we need to stop thinking about those things. Controlling the location of the bits is not what matters. Features and performance are what matter. Results in the real world matter. Look at the systems available and decide what blend of features fits your organization and workload best.

Full disclosure: My company provides storage consulting on all of the platforms discussed above. We sell NetApp, Sun, and HDS products. We do not sell EMC products.

Share this post: