S3 performance and partitioning
Getting throughput from S3. This post digs into the topic, drawing on recent discussion from Hacker News, Reddit, and the wider community.
The conversation that prompted this post is linked in the sources. I have tried to capture the substance of the discussion rather than just summarize it, because the comments are usually where the real insight lives.
What happened
Tagging is boring and essential. Without consistent tags, cost attribution is guesswork.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
Cloud cost is the new technical debt. Teams that do not track spend by service and team end up with bills that nobody can explain and nobody can reduce.
Networking is where cloud bills hide. Data transfer charges between regions and out to the internet add up fast.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
Spot instances are free money if your workload can tolerate interruption. Most batch and async workloads can.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
Networking is where cloud bills hide. Data transfer charges between regions and out to the internet add up fast.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
Cloud cost is the new technical debt. Teams that do not track spend by service and team end up with bills that nobody can explain and nobody can reduce.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
Observability is the difference between knowing something is broken and knowing why. Invest in it before you need it.
Infrastructure as code is not optional at scale. Clicking in the console is fine for learning and unacceptable for production.
Managed databases trade operational burden for cost and flexibility. Know what you are giving up.
Why it matters
Serverless is great until it isn't. Cold starts, concurrency limits, and vendor lock-in are real costs that show up at scale.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
Networking is where cloud bills hide. Data transfer charges between regions and out to the internet add up fast.
Autoscaling is not a substitute for capacity planning. It reacts to load; it does not predict it.
Reserved instances and savings plans reward commitment. If your workload is stable, not using them is leaving money on the table.
Cloud cost is the new technical debt. Teams that do not track spend by service and team end up with bills that nobody can explain and nobody can reduce.
Spot instances are free money if your workload can tolerate interruption. Most batch and async workloads can.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
Reserved instances and savings plans reward commitment. If your workload is stable, not using them is leaving money on the table.
Observability is the difference between knowing something is broken and knowing why. Invest in it before you need it.
Networking is where cloud bills hide. Data transfer charges between regions and out to the internet add up fast.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
Serverless is great until it isn't. Cold starts, concurrency limits, and vendor lock-in are real costs that show up at scale.
Practical takeaways
- Serverless is great until it isn't.
- Networking is where cloud bills hide.
- Multi-region is not multi-cloud.
- Autoscaling is not a substitute for capacity planning.
- IAM is the foundation of cloud security.
- Infrastructure as code is not optional at scale.
- IAM is the foundation of cloud security.
- Networking is where cloud bills hide.
- Cloud cost is the new technical debt.
- The shared responsibility model means the cloud secures the infrastructure and you secure everything else.
Autoscaling is not a substitute for capacity planning. It reacts to load; it does not predict it.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
Networking is where cloud bills hide. Data transfer charges between regions and out to the internet add up fast.
Tagging is boring and essential. Without consistent tags, cost attribution is guesswork.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
Spot instances are free money if your workload can tolerate interruption. Most batch and async workloads can.
Managed databases trade operational burden for cost and flexibility. Know what you are giving up.
Managed databases trade operational burden for cost and flexibility. Know what you are giving up.
Backups are not disaster recovery. Test your restores or you do not have backups.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
Cloud cost is the new technical debt. Teams that do not track spend by service and team end up with bills that nobody can explain and nobody can reduce.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
Reserved instances and savings plans reward commitment. If your workload is stable, not using them is leaving money on the table.
Observability is the difference between knowing something is broken and knowing why. Invest in it before you need it.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
Backups are not disaster recovery. Test your restores or you do not have backups.
Reserved instances and savings plans reward commitment. If your workload is stable, not using them is leaving money on the table.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
Sources
This post is informed by discussion on docs.aws.amazon.com and related threads on Hacker News and Reddit. Primary source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html.
Tagging is boring and essential. Without consistent tags, cost attribution is guesswork.
Cloud cost is the new technical debt. Teams that do not track spend by service and team end up with bills that nobody can explain and nobody can reduce.
Infrastructure as code is not optional at scale. Clicking in the console is fine for learning and unacceptable for production.
Managed databases trade operational burden for cost and flexibility. Know what you are giving up.
Autoscaling is not a substitute for capacity planning. It reacts to load; it does not predict it.
Managed databases trade operational burden for cost and flexibility. Know what you are giving up.
Observability is the difference between knowing something is broken and knowing why. Invest in it before you need it.
Tagging is boring and essential. Without consistent tags, cost attribution is guesswork.
Autoscaling is not a substitute for capacity planning. It reacts to load; it does not predict it.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
Cloud cost is the new technical debt. Teams that do not track spend by service and team end up with bills that nobody can explain and nobody can reduce.
Observability is the difference between knowing something is broken and knowing why. Invest in it before you need it.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
Observability is the difference between knowing something is broken and knowing why. Invest in it before you need it.
Serverless is great until it isn't. Cold starts, concurrency limits, and vendor lock-in are real costs that show up at scale.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
Cloud cost is the new technical debt. Teams that do not track spend by service and team end up with bills that nobody can explain and nobody can reduce.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
Backups are not disaster recovery. Test your restores or you do not have backups.
The shared responsibility model means the cloud secures the infrastructure and you secure everything else. Most breaches are in the everything else.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
Autoscaling is not a substitute for capacity planning. It reacts to load; it does not predict it.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
IAM is the foundation of cloud security. Get it wrong and nothing else matters. Get it right and most attacks become much harder.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
Reserved instances and savings plans reward commitment. If your workload is stable, not using them is leaving money on the table.
Managed databases trade operational burden for cost and flexibility. Know what you are giving up.
IAM is the foundation of cloud security. Get it wrong and nothing else matters. Get it right and most attacks become much harder.
Managed databases trade operational burden for cost and flexibility. Know what you are giving up.
IAM is the foundation of cloud security. Get it wrong and nothing else matters. Get it right and most attacks become much harder.
Reserved instances and savings plans reward commitment. If your workload is stable, not using them is leaving money on the table.
Backups are not disaster recovery. Test your restores or you do not have backups.
Spot instances are free money if your workload can tolerate interruption. Most batch and async workloads can.
Networking is where cloud bills hide. Data transfer charges between regions and out to the internet add up fast.
Networking is where cloud bills hide. Data transfer charges between regions and out to the internet add up fast.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
Tagging is boring and essential. Without consistent tags, cost attribution is guesswork.
The cloud is just someone else's computer, and that someone else is very good at charging you for convenience. The question is always whether the convenience is worth the markup.
Infrastructure as code is not optional at scale. Clicking in the console is fine for learning and unacceptable for production.
Cloud cost is the new technical debt. Teams that do not track spend by service and team end up with bills that nobody can explain and nobody can reduce.
Backups are not disaster recovery. Test your restores or you do not have backups.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
IAM is the foundation of cloud security. Get it wrong and nothing else matters. Get it right and most attacks become much harder.
Observability is the difference between knowing something is broken and knowing why. Invest in it before you need it.
Infrastructure as code is not optional at scale. Clicking in the console is fine for learning and unacceptable for production.
Spot instances are free money if your workload can tolerate interruption. Most batch and async workloads can.
Cloud cost is the new technical debt. Teams that do not track spend by service and team end up with bills that nobody can explain and nobody can reduce.
Serverless is great until it isn't. Cold starts, concurrency limits, and vendor lock-in are real costs that show up at scale.
Networking is where cloud bills hide. Data transfer charges between regions and out to the internet add up fast.
IAM is the foundation of cloud security. Get it wrong and nothing else matters. Get it right and most attacks become much harder.
Serverless is great until it isn't. Cold starts, concurrency limits, and vendor lock-in are real costs that show up at scale.
Spot instances are free money if your workload can tolerate interruption. Most batch and async workloads can.
Tagging is boring and essential. Without consistent tags, cost attribution is guesswork.
Infrastructure as code is not optional at scale. Clicking in the console is fine for learning and unacceptable for production.
Multi-region is not multi-cloud. Running in two AWS regions is redundancy. Running on AWS and GCP is strategy, and it is expensive.
Infrastructure as code is not optional at scale. Clicking in the console is fine for learning and unacceptable for production.
Reserved instances and savings plans reward commitment. If your workload is stable, not using them is leaving money on the table.