# NetApp E-Series sizing for Splunk 9 with SmartStore Some rules of thumb that can help you size E-Series for Splunk better - [Introduction](#introduction) - [Existing material](#existing-material) - [Related posts on this blog](#related-posts-on-this-blog) - [Observations](#observations) - [TB per day](#tb-per-day) - [Events per day](#events-per-day) - [Sizing challenges](#sizing-challenges) - [Splunk 9](#splunk-9) - [SmartStore](#smartstore) - [StorageGRID](#storagegrid) - [StorageGRID performance](#storagegrid-performance) - [StorageGRID capacity](#storagegrid-capacity) - [MinIO](#minio) - [E-Series storage layout](#e-series-storage-layout) - [Storage sizing example with Splunk Enterprise with SmartStore](#storage-sizing-example-with-splunk-enterprise-with-smartstore) - [SmartStore on flash media](#smartstore-on-flash-media) - [SmartStore on NL-SAS](#smartstore-on-nl-sas) - [SmartStore churn and slowness](#smartstore-churn-and-slowness) - [Multi-array and multi-site sizing](#multi-array-and-multi-site-sizing) - [What's still missing](#whats-still-missing) - [Conclusion](#conclusion) - [Appendix A - Anecdotal Figures](#appendix-a---anecdotal-figures) ## Introduction In recent years NetApp published several Splunk-related Technical Reports (TRs), but they're not all fresh and/or easy to digest. This post is an attempt to highlight the useful parts of existing TRs and add some new angles. ## Existing material I am aware of the following material related to Splunk with E-Series (Splunk "Classic") or StorageGRID (Splunk with SmartStore). - [NetApp HCI and Splunk Enterprise Solution with Arrow](https://www.netapp.com/media/17054-tr4778.pdf) - no longer sold, but well-done, and may be useful to NetApp HCI or SolidFire (even E-Series) users - [NetApp EF600 with Splunk Enterprise](https://www.netapp.com/pdf.html?item=/media/72003-tr-4930.pdf) - Splunk Classic with NetApp EF600 (fastest) - [NetApp EF300 with Splunk Enterprise](https://www.netapp.com/media/57104-tr-4903.pdf) - Splunk Classic with NetApp EF300 (fast) - [NetApp StorageGRID with Splunk SmartStore](https://docs.netapp.com/us-en/netapp-solutions/data-analytics/stgr-splunkss-introduction.html) - SmartStore-related ## Related posts on this blog - [Virtualized Splunk on NetApp HCI and EF Series iSCSI storage](/2020/12/31/virtualized-splunk-on-netapp-hci-and-ef-series.html) (2020) - [Virtualized StorageGRID for Splunk SmartStore on NetApp HCI and EF-Series array](/2021/01/15/netapp-hci-storagegrid-splunk-smartstore-on-efseries.html) (2021) - about StorageGRID in VMs ## Observations ### TB per day The first of my Splunk-related blog posts was an attempt to evaluate EF280 (predecessor of EF300) with Splunk while not actually using Splunk because I couldn't. I did this because the first NetApp TR above (HCI + SolidFire) didn't use E-Series for Hot/Warm. I used just one NetApp HCI server with Hot/Warm Tier on VMFS backed by EF280 and executed (presumably) Splunk-like workloads in that environment. From that post: > EF280 with one expansion shelf (so that the controller shelf and one expansion shelf occupy 4 rack units and have 24 SSDs each) is slightly faster than 7-8 TB/day - let’s say it’s good for 10 TB/day All I had was an ESXi server connected to an EF280 - so I could run Bonnie++ but not much else at the time - so that was an educated guess before NetApp released E/EF Series-based TRs. How close was I? The EF300/EF600 TRs didn't come until later (EF300 in 2021, EF600 in 2022), but the first TR says: > The ingested data was then scaled to 4TB per day during scale testing. Unfortunately, there's nothing in that TR that makes me believe they hit the array maximum - not a single chart shows where the performance bottleneck was (or indeed, whether it was discovered at all, which is information I would expect to see in such a TR). The EF600 TR: > The ingest was then scaled to 870GB of data per indexer, for a new total of 10TB per day. So, they scaled ingress performance to 4TB and 10TB per day, using EF300 and EF600, respectively. But we still don't know whether either of those maxed out the array, and for storage sizing purposes that doesn't help. If we want to index 5TB/day and our budget is tight, should we buy an EF300 for that? No way to tell. As another reference point, that HCI/SolidFire TR at the top claims that 1 GB/s in write storage performance roughly (there's a range) translates to 1 TB per day in indexing performance. Neither the TR nor I suggest we should use just that one rule (it always depends, as they say), but EF300 can write at 7 GB/s so I think my 10TB/day guesstimate is closer to the truth than the 4TB figure from the TR. ### Events per day For this we'd have to know what kind of events, etc. Something [like this](/2023/02/25/elasticsearch-eseries-performance.html#siem-indexing-workload) or better. Unfortunately, I haven't had a chance to do this myself with Splunk, and the EF-Series TRs didn't do it either. All we know is they ingested some "logs". That doesn't help us with sizing based on number *and* type of events. ### Sizing challenges As the EF-Series TRs show, log type (firewall, Web app, etc.) and other details (index structure, field parsing, transformations, etc.) matter and that's why Splunk and their practitioners distinguish among them. Depending on those differences, the same number of "GB/day" may take a shorter or longer time to index, may compress better or worse, and may result in faster or slower searches. But the TRs assume they compress exactly the same (85%) and that indexes always equal roughly 65% of source volume, so knowing that a node can ingress "870 GB/day" doesn't really tell you much, neither in terms of performance nor in terms of capacity. If 870 GiB of logs per indexer comes in, that may require 870 GB x (0.15 + 0.35) of storage (x 2 for the second replica), but it can also be 30% more if your logs aren't very "average". That also means storage *performance* (not just capacity!) required to accommodate that capacity may be $PERF GiB/s or ($PERF x 1.3) GiB/s, depending on the logs and indexes. On the one hand, any example would be different from what most customers have so a detailed estimate still has to be done [by each customer for their own data](https://docs.splunk.com/Documentation/Splunk/9.1.1/Capacity/Estimateyourstoragerequirements#Use_a_data_sample_to_calculate_compression). On the other hand, it would at least give us some idea of how to size for at least one specific workload. The TRs should have provided examples of logs that were used, so that everyone is on the same page. But that didn't happen. The first TR (HCI with SolidFire storage) has more details about the data and workloads used to estimate performance and capacity requirements, but it wasn't done with E-Series (E-Series was used for Cold Tier, without performance sizing). ## Splunk 9 It doesn't seem that Splunk updates their [sizing recommendations](https://docs.splunk.com/Documentation/Splunk/9.1.1/Capacity/Summaryofperformancerecommendations#) for storage because for v9.1.1 they look *exactly* the same like they do in their earliest documentation that's still online, [v7.0.0](https://docs.splunk.com/Documentation/Splunk/7.0.0/Capacity/Summaryofperformancerecommendations)! It's funny that the recommendation table doesn't even go beyond 3 TB/day, which was a lot when v7 came out but as we saw from the NetApp TRs, even an EF300 is capable of more than that. Both EF300 and EF600 performance is literally "off the charts" because Splunk hasn't updated their charts since v7.0.0 (click and compare!). I suspect they no longer update it because when they started everyone thought they were smart and almost exclusively used DAS (clearly, no lessons were learned from Hadoop's HDFS debacle in the enterprise), so storage sizing like that made some sense. But Splunk users discovered the benefits of compute-storage disaggregation and it was no longer possible to create such simplistic sizing charts. For example, there's no need to make 3 copies on a *protected, redundant* external storage array but if you don't do that, then your sizing will be quite different from DAS sizing as you will need fewer indexers. Furthermore, enterprise storage offers sophisticated configuration options, from RAID levels to scale out and whatnot, so today it's no longer possible to make a "simple" chart on how to do it right. But that's not a good excuse to continue publishing completely outdated recommendations either. To make things even more complicated for performance and capacity sizing, since v7 SmartStore allows users to size Hot and Cache Tier primarily for performance, and use S3 primarily for capacity, which means Splunk would be sized very differently for SmartStore deployments than without them. I do understand those challenges prevent the creation of a simple sizing chart and that's why I can't write just one post on this topic, but still... It could be done better. Maybe Cisco will come up with detailed sizing for storage used by UCS. ## SmartStore The official SmartStore sizing is also ambiguous, just like the generic Spunk sizing charts. But, long story short: - At the minimum, SmartStore must accept uploads (PUTs) and at 10 TB per day that's (using generic assumptions) 10 TB x (0.15 + 0.35) = 5 TB PUTs per day. That's less than 100 MB/s - easy to handle! Splunk doesn't need to make multiple copies (replicas) on S3, but object storage makes multiple copies on its own, or uses EC to prevent downtime and/or data loss - For searches, if we don't rely on most searches being served from SmartStore cache, they'd surely be slow when served from SmartStore on NL-SAS, and not very fast even with flash-based object storage. But that's why SmartStore Cache Manager exists - it caches data that's recent and/or regularly searched, and pulls data from SmartStore on demand when data it needs is not in cache. We do need to size it appropriately for our needs. For Splunk Enterprise, 7-10 days in cache is recommended (without replicas, as a copy or copies remain(s) in object store). - SmartStore object cache is retained on servers and those disks don't necessarily need to be on E-Series - they can be local to indexers in which case you wouldn't need E/EF Series. But Hot Tier and Cache can be on E/EF, it may be cheaper and fast enough. Example: 12 indexers with cache on 2 disks in internal RAID 1 = 24 disks. With EF-Series we may be able to avoid overprovisioning and accomplish the same with 16 SSD disks in a single 2U controller shelf. The StorageGRID SmartStore TR at the top also places Hot/Cache data on EF600 although it fails to highlight any details that would have been useful for performance sizing for SmartStore Cache on E/EF-Series... - With indexer clusters, site replication factor and search replication factor must be the same (for example, 3/3 or 2/2) (see [Splunk documentation](https://docs.splunk.com/Documentation/Splunk/9.1.1/Indexer/AboutSmartStore)) Generally speaking if SmartStore cache is sized appropriately, SmartStore data can be on NL-SAS - PUTs and GETs will be steady and predictable, as most search queries will be satisfied from SmartStore cache. That's how StorageGRID was configured in the StorageGRID TR, which use SG6060 with NL-SAS media for object data volumes. Workload-wise, I think it's reasonable to assume SmartStore cache is a predominantly-read workload and since writes to Object Store and SmartStore Cache Tier are sequential, Cache Tier can be on SSD RAID 6 volume groups (or RAID 6 on DDP) - there's no significant re-write penalty and the convenience of sharing DDP by using RAID 6 is high (compared to using dedicated RAID 5 volume groups for SmartStore cache volumes, for example). I've no idea what the average SmartStore workload looks like (one would hope Splunk and each Splunk-certified vendor would have a technical deep-dive about it - wrong!), but I estimate it's >80% GET and <10% PUT with a small SmartStore Cache churn when cache capacity is sized correctly. The other 10% is various STAT and DELETE stuff. ### StorageGRID #### StorageGRID performance A SmartStore performance test with Splunk Enterprise v7 is available [here](https://docs.netapp.com/us-en/netapp-solutions/data-analytics/stgr-splunkss-splunk-architecture.html). Using 3 x SG6060, they approximately 1 GB/s in PUTs per each StorageGRID SG6060 node. GETs are obviously easier, but more on that further below. That means we can go with close to 1 GB/s per SG6060 for PUTs for a correctly sized SmartStore cache. That's much more than is necessary for the continuous migration of data from Hot Tier to SmartStore in mid-sized Splunk clusters. As we saw above, 10 TB/day translates into less than 100 MB/s in PUTs, so there would be plenty of GET performance left over. SmartStore performance in Splunk v9 is likely somewhat better than it was in v8 or v7, and StorageGRID can scale to many nodes, so additional throughput can be achieved by simply adding SG or SGF (all-flash) storage nodes to the (StorageGRID) cluster. VM-based StorageGRID with a E-Series back-end is likely to be somewhat slower in terms of "per node" performance and also more difficult to manage, but it's available for users with smaller requirements or edge sites where both Splunk and StorageGRID runs in VMs. See my post on StorageGRID VMs (link at the top) for more on this. StorageGRID S3 clusters, physical or virtual, can be stretched across sites and optionally set to strong bucket consistency, if you value data integrity and availability at cost of write performance. If you look at the StorageGRID performance tests at that link, read performance (GETs) was much higher, but high GETs would be needed only when rare searches that go beyond the cache hit non-cached index data ("buckets") and when cache naturally expires, so it would be the last 5% or 1% of searches where "maximum" read performance matters. It would have been nice if the TR deep-dived into that - given that PUTs are trivial to satisfy even with the smallest SG6060 cluster, so performance sizing greatly depends on how many MB/s in GETs we really need in some common scenarios (say, at least one documented SIEM search scenario with x% of searches missing SmartStore cache). If you need a two- or three-site deployment or frequently search out-of-cache buckets, it's best to engage a specialist to discuss your requirements. Even Splunk has several approaches to multi-site searches, and StorageGRID has several approaches as well. It's hard to create generic guidelines for that. #### StorageGRID capacity Capacity-wise, Splunk stores one copy on Object Store, and Object Store takes care of its own redundancy. Three StorageGRID appliances would use RF2 on top of RAID/DDP, and four could use EC2+1 (what MinIO calls EC:1) on top of RAID/DDP. By simply looking at the StorageGRID product pages we can see that each SG6060s has 58 disks and the smallest disk currently available being 4 TB. That's almost 700 TB (58 disks x 3 x 4TB) of raw capacity, so if "usable after RAID" is around 600 TB, the smallest 3-node cluster would be 300 TB after 2-copy protection on StorageGRID. SG6060s have a high-density option with 1 or 2 expansion shelves (each with 60 NL-SAS disks) in the case you need add capacity but not performance. For less than that and with slow data growth expectations, we may want to consider StorageGRID in VMs instead. For 100-300 TB we'd still use NL-SAS, but for less than 100 TB it may be cheaper to just use SSDs for everything (Hot/Warm, SmartStore). If you wonder why would anyone do that when they could deploy Splunk without SmartStore, the reason is some organizations made a decision to deploy SmartStore and as SmartStore capacity grows, StorageGRID can be expanded with NL-SAS nodes, and SSD-based VMs can be later removed from the cluster - that's not more complicated than enabling SmartStore at a later time. ### MinIO Another option for smaller, single site SmartStore object stores - not promoted by NetApp, but available for users who want to use it - is MinIO. As NetApp doesn't validate or promote this as its solution for Splunk, we have no official details to go by. However, we know that: - Virtualized EF600 or EF300 can easily give you a [few GB/s per second](/2022/10/21/minio-performance-netapp-e-series.html) - MinIO's [Erasure Coding can be used](/2023/09/03/minio-erasure-coding-and-netapp-e-series.html#single-rack) with E/EF-SEries, but it's not mandatory to fully follow MinIOs' EC recommendations that are based on DAS and JBOD. It's possible to run four MinIO VMs or containers with 1 volume per server and EC:1, for example. MinIO considers that to be a "reduced redundancy" configuration and doesn't officially support it (they assume JBODs), but is fine for MinIO with E-Series: - Volumes assigned to MinIO are protected by E-Series RAID or DDP - A disk may fail without impacting MinIO service - A controller may fail while still providing > 5-10 GB/s (depending on model) - A path to controllers or enclosures may go down without making performance drop below 5-10 GB/s (depending on model) - Any MinIO node can fail without impacting availability or data integrity (EC:1 such as EC 3+1) - E/EF Series' storage protection overhead would be around 1.25x for RAID6 and with 1.5x for EC:1 (3+1 for a 4-node, 4 volume cluster) on top of that total overheads from E/EF Series and MinIO would be (1.5 x 1.33) As mentioned above, this could use NL-SAS media. Aside from that post with an older test run, I tried a SmartStore-like workload test with a single MinIO server. Assumptions: - 8, 16 and 32 threads with 750 MB objects with the following workload mix - 80% GET - 10% PUT - 5% STAT - 5% DELETE - S3 performance tested with Warp (multipart PUTs were not used; I don't think they would matter much) - Latest MinIO version with 4 volumes EC:1 and EC:0 (both were evaluated) - Ubuntu 22.04 with iSER with all paths on a single controller - EF570 with RAID 1-style volumes in DDP (16/2) with SAS SSD media ![MinIO with EC:0 on four E-Series RAID1/DDP volumes](/assets/images/eseries-minio-erasure-coding-02.png) The dots represent performance of each individual volume, and stack height is total (aggregate) performance of storage volumes used by MinIO. (I use [EPA](/2023/11/04/eseries-perf-analyzer-epa-330.html) to visualize E-Series performance). I estimate on-premises SmartStore workloads that can't justify dedicated object stores would not have more than 32 concurrent PUT/GET requests. I also tried 64 and it performed about the same as 32. Two or four concurrent PUT or GET requests on each of 16 indexers would be required to hit 32 or 64 parallel requests, and I think that would be rare with properly sized SmartStore Cache on indexer nodes. Of course, we could deploy four MinIO nodes for more performance, but maybe we could get away with just one (if we didn't mind occasional short-lived timeouts when the VM/container is failed over or restarted, or MinIO upgraded.) Both EC:0 and EC:1 were tested and the difference between EC:0 and EC:1 observed from several runs was small (it's hard to tell from the chart which is EC:0 and which EC:1). Given the same storage throughput, EC:0-based service should perform better on the front end because of no EC overhead for writes, but that wasn't even readily noticeable which suggests we weren't close to hitting any limit. With more servers we would be able to eventually saturate storage (IO paths or controllers or disks), but for smaller environments the performance penalty of EC:1 is insignificant and with 4 volumes the main "cost" is 33% more capacity (assuming EC 3+1). Because of the large object sizes, NL-SAS should perform similarly well. Even with multipart uploads, objects would still be large. ## E-Series storage layout Using DDP (disk pools, rather than traditional RAID) is the easy choice. I would recommend classic RAID only for large environments which wouldn't require over-provisioning due to multiple disk groups and would benefit from 5 or 10% more performance thanks to micro-management of RAID configuration and storage layout. DDP must use same disk type and ought to use disks of the same size, so for a single all-flash DDP we'd have to use all NVMe or all SAS flash disks of the same capacity. For hybrid setups (SSD for Hot/Cache, NL-SAS for SmartStore SDS) we'd use two DDPs, one for each disk type and size. Hot/Cache Tier could use RAID 6-type volumes on DDP, but we can use RAID-1 as well. RAID-1 volumes provide a better performance at the cost of additional capacity (100% more vs. ~25% more for RAID 6). But with SmartStore Hot/Cache Tier could be small enough so that extra capacity requirements wouldn't bother us much: 10 days of 5 TB/day * RF2 = 100TB usable or 200 TB in RAID 1 capacity, compared to some 125TB raw for RAID 6. If you need performance, use RAID 1 on DDP, if capacity, use RAID 6 on SSD-based DDP. Cold or (for SmartStore users) SmartStore volumes could use NL-SAS on RAID 6-on-DDP or classic RAID 6 volume groups. For those new to E-Series, the upper box is an E/EF-Series 2U controller shelf with 24 2.5" disk slots (commonly, and on EF Series exclusively, populated with SSDs). ![Splunk with SmartStore and E-Series](/assets/images/eseries-smartstore.png) The box at the bottom is DE460C, a [4U expansion enclosure](https://www.netapp.com/data-storage/e-series/e5700/) that can accommodate up to 60 3.5" drives (commonly 4-22 TB NL-SAS) and multiple enclosures can be attached to each controller. There are many ways to do this, obviously. Depending on how SmartStore (depicted at the bottom) is designed, it could range from 1 VM with 1 big R6 volume on DDP (in a MinIO VM or K8s-based container, for example) to many nodes and many volumes with Erasure Coding (supported by both StorageGRID and MinIO) on top of protected storage volumes provided by E-Series. ### Storage sizing example with Splunk Enterprise with SmartStore Splunk recommends 30 days for Splunk Enterprise and 90 days for SIEM (Enterprise Security). Assuming Splunk Enterprise with 10 TB of logs per day (achievable with EF600 and likely EF300) that's 5 TB of new data per day. Given RF2 for Hot, RF1 for search cache, 30 days of retention and SmartStore being somewhere else: - 7 days of Hot * RF2 * 5 TB = 70 TB - (30-7) days of SmartStore Cache * 5 TB = 23 * 5 TB = 115 TB This [Splunk community post](https://community.splunk.com/t5/Knowledge-Management/SmartStore-Sizing/m-p/453360) by a reputable community member uses that formula for Splunk's cluster-level Hot/Cache Tier: ```raw Global Cache = Daily Ingest Rate x Compression Ratio x (RF x Hot Days + (Cached Days - Hot Days)) Cache sizing per indexer = Global Cache sizing / No. of indexers # My translation to "spreadsheet" language # Global Cache sizing = DailyTB *(((1-CompressionSavingsPct)/1)+((1-IndexSizeAsPctOfSource)/1))*(RFFactor*HotDays+(AllDays-HotDays)) # Example: 10 TB/day, 85% compression savings, 35% index size, 7 days of Hot with RF2 and 30-7 of SmartStore Cache # 10TB x ((1-0.85)/1) + ((1-65%)/1) x (2 x 7 + (30 - 7)) = 185 TB # Cache sizing per indexer = Indexers / Global Cache # If 8 indexers = 185 TB / 8 = 23.125 TB per indexer ``` Add some capacity for various overheads, say 20-25%. Add more if your environment is growing quickly or you have absolutely no clue about compression and index sizes in your environment. In an earlier version of this post (it's quite long so I've edited it a bunch of times) I noted how Splunk Enterprise users with Cold Tier can move to SmartStore and enjoy the benefits of Cold Tier because Cold Tier continues to be used as cache landing zone for recalled data from SmartStore. Which means you can have something like: - RAID 1 on SSDs for Hot - RAID 6 on SSDs for Cold But that can't work for new deployments where Cold Tier is being eliminated. Says Splunk: > The concept of cold buckets goes away, because the need to distinguish between warm and cold buckets no longer exists. Oh, really? But that's not true. It would be beneficial if we could continue using Hot and Cold Tiers on volumes based on a different RAID level such as RAID 1 and RAID 6. But Splunk is still designed with DAS and JBODs in mind so they don't see that and you don't get to choose. Because of that new deployments with SmartStore stuff all local data to a uniform Hot/Cache Tier. ### SmartStore on flash media Since SmartStore Cache should be on SSDs and big enough to keep 30 days of cache, why not just put SmartStore data on a flash-backed object store and run Splunk with a small local SmartStore cache? Something like this: | Setup | Cache size| Cache media | SmartStore size | SmartStore media | |-------------|-----------|-------------|-----------------|------------------| | Mainstream | 150 TiB | SSD | 300 TiB | NL-SAS | | Niche | 50 TiB | SSD | 300 TiB | SSD | I hope this table makes the idea clear. Rather than having a fully-sized 30d cache and a separate NL-SAS object store for 45-60d (how many that really depends on object store configuration), what if we configured a smaller cache and eliminated NL-SAS? Most searches are for 7-10 days old data, so we could shrink SmartStore cache on the account that object store should be much faster now and able to withstand significant churn. But something like that would require evaluation and testing. Ain't nobody got time for that! ### SmartStore on NL-SAS SmartStore on NL-SAS is easy - as long as E/EF Series controllers haven't been maxed out, we can simply size RAID 6 or DDP for sequential performance to come up with the right number of NL-SAS disks. ### SmartStore churn and slowness How to prevent SmartStore churn: - Aim for a reasonable number of concurrent downloads from SmartStore: this should be relatively low if local cache is large enough. If it's not, this could be high (say, 5 per indexer at all times) and result in slow searches. - Amount of data downloaded by SmartStore: same as above - it should be stable and in accordance with expectations. If an 8-indexer cluster download 30 TB of S3 data per day, maybe there are bad search queries or local cache capacity is undersized? SmartStore churn can also happen if object store is simply too slow (undersized in the sense of being unfit for your requirements from Day 1, or maybe unable to keep up with increasing workload). If that's the main problem, you'd need to boost its performance. - StorageGRID appliances: add nodes to grow capacity and performance - StorageGRID VMs or other object store on E-Series: - Add disks to E-Series DDP (usually low number of NL-SAS disks in a group or pool is the main reason) - Add nodes to object store cluster (if the problem is CPU or network saturation on the nodes) - Add arrays (least desirable, but unavoidable if controllers' performance is insufficient) ## Multi-array and multi-site sizing Multi-array sizing could use two or three arrays. Simply split the performance among multiple arrays. If your original sizing resulted in one EF600, a two- or three-array design could use EF300. Individual arrays may be sized asymmetrically, and even heterogeneous designs can work (all-flash for site A, all-NL-SAS for site B), but I'd suggest to talk to someone experienced with these things - both from Splunk and NetApp. ## What's still missing The free Splunk license only works for all-in-one deployments and is capacity-limited which makes it hard to perform tests that are more meaningful than educated guessing. They do work, but it's not very convenient. But this is a list of things I'd still like to do in the future: - Events per second per indexer for common log formats with publicly available data sets - Storage performance characterization for Hot Tier - Storage performance characterization for Cache Manager with high cache hit rate (say 95%) - Performance comparison between RAID 1 and RAID 6 on E-Series DDP for Hot and SmartStore cache - Workload characterization of SmartStore (especially request sizes, concurrency, breakdown by type of operation) ## Conclusion The official E-Series and EF-Series TRs don't go into enough detail related to sizing, and while the SmartStore TR is more detailed it's for StorageGRID and not E/EF-Series. That means E-Series users may need a bit of extra help. Thanks to the old and more detailed NetApp HCI Splunk TR and various anecdotal evidence and unofficial information, we can make more educated guesses regarding E-Series performance for Hot data, SmartStore cache, and VM-based object storage for SmartStore. I think these are reasonably conservative guesstimates that you can use when sizing storage for single-array deployments. ![Guesstimate for Splunk Enterprise 9 with SmartStore and E-Series](/assets/images/eseries-smartstore-guesstimate.png) This doesn't mean it's guaranteed to work and that your data would cooperate even if these guesstimates were correct for the average user. I hope this post gives additional actionable information to those interested in using NetApp storage with Splunk Enterprise with SmartStore. ## Appendix A - Anecdotal Figures After publishing this post I managed to run some simple tests using Splunk 9.1.1 (stand-alone, **no replicas**) and two kinds of data. - Ubuntu 22.04 syslog file: 197 MB file was 35 MB post-indexing. Even after 2x replication, this would still be barely 30% of the original data size, so it's suspiciously low. - Kubernetes performance metrics (time series data with various tags): 120 GB file containing 26 million JSON "events" indexed (index + raw data) to 31% of its original size. But after 2x replication this would be 62% of its original size - Index reduction was not enabled (default) because it's off by default and not supported for SmartStore The first file was too small for meaningful analysis, but the second was large. This screenshot is from when it was still being processed. ![Splunk Enterprise 9 indexing time-series Kubernetes metrics on EF570](/assets/images/eseries-smartstore-k8smon-tsdb.png) It was indexed at 75 MB/s per second with average CPU utilization 21% of 48 vCPUs (24 3 GHz cores with HT on). Splunk wrote in "waves" oscillating between 0 and 60 MB/s, with the average was around 23 MB/s which makes sense given total index size was 31% of original data (75 MB/s * 0.31). Number of requests-wise vast majority was small (4-8 KB) and less than 10% were large, resulting in the "average" request size of over 30 KB. If we extrapolate to RF2 (50 MB/s write performance for 75 MB/s of logs) it appears a single EF-Series EF300 should be able to easily index 10 TB per day and still serve search requests. 50 MB/s * 86400 = 4.2 TB per day with one indexer. I can't explain the performance discrepancy with the EF300 TR. Some of the advantage from my testing may be due to a better compression or newer Linux OS, but like I said above I still believe the TR got nowhere near the EF300 performance limit. I can't think of anything else. Likewise, 16 indexers indexing at this rate would need just 400 MB/s in PUT performance per second (16 x 25 MB/s). Object store that supports 4 GB/s of mixed performance would need a small fraction - 10% - for PUTs. Anyway, as such details are missing from the TRs and I don't have a Splunk lab at my disposal to give it a try, we can only extrapolate like this.