Skip to main content

Biggest project management mistake

This holds true for developers, project managers, entrepreneurs, everybody who has shipped software to the real world. The biggest mistake one can make while shipping software is not shipping buggy code or half-baked features (and they are all easy to uncover given sufficient QA), or not planning for failure. The biggest mistake that folks make is - not planning for success.

Trust me - shit breaks loose when your app gets hot over the night! So here's a list of concerns one needs to plan for, before launching a site:

Cache everything - images, javascript, style sheets, html templates - everything. I use nginx at the frontend proxy layer and it does a full page caching for entire html pages. Works like a charm!
At the application layer, use Redis as a TTL-based cache.

Deliver static content really fast
Images, JS, stylesheets - they need to be rendered by a specialized static files host - a CDN in essence. Use a managed service provider (Amazon CloudFront), or deploy nginx (with local caching) over a separate box and serve yourself.

Scaling load-balancers/frontend proxies
Plan your deployment in such a way that adding and removing first layer application servers should be ridiculously simple. This means a load-balancer is an absolute must. Here again - you can use Amazon ELB or deploy nginx or haproxy.  But if you happen to deploy a load-balancer yourself, keep in mind that servers die all the time and you don't want your release sabotaged by a dead load-balancer. Plan for failure. What happens when the load-balancer dies? You spin up a new one! And that should be ridiculously simple - use AMIs, docker, vagrant, puppet, chef, anything.

Scaling application servers
Keep a couple of application servers in standby so that when the load comes in, you can just plow them in quickly. A better strategy is to use docker or other packaging tools which let you spawn applications quickly.
Use event-based I/O at application server layer. For python (flask, django) applications, I use uwsgi as the app container. Gunicorn is also a good choice.

Scaling Databases
Index your tables and views correctly. Shard. This machine needs to be high on memory. Increase max parallel connections setting before going live.

Logging and alerts
You cannot improve something you cannot measure. And your users mailing you about app crashes is not the best idea of usability testing. Use NewRelic agents on your web server components, use Crashlytics in your mobile app. If hell is breaking loose - you should be the first to know, not your users. ELK and ElastAlert make a good choice too, but for beginners, a NewRelic based alerting mechanism is good enough. Amazon users can very well use CloudWatch.

This is part one of the post. In part two, I will talk about:
  1. Automation and tests
  2. Blue-green deployments
  3. Reporting
  4. Backups
  5. Logging


  1. I found this blog in search results while i am searching for aws jobs in hyderabad. Thank You admin sharing for this information. Which is useful for me.

  2. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java Training in Chennai. or learn thru Java EE Online Training from India . Nowadays Java has tons of job opportunities on various vertical industry.


Post a Comment

Popular posts from this blog

Capture and compare stdout in python unit tests

A recent fan of TDD, I set out to write tests for whatever comes my way. And there was one feature where the code would print messages to the console. Now - I had tests written for the API but I could not get my head around ways to capture these messages in my unittests. After some searching and some stroke of genius, here's how I accomplished capturing stdout.

On working remote

The last company I worked for, did have an office space, but the code was all on Github, infra on AWS, we tracked issues over Asana and more or less each person had at least one project they could call "their own" (I had a bunch of them ;-)). This worked pretty well. And it gave me a feeling that working remote would not be very different from this. So when we started working on our own startup, we started with working from our homes. It looked great at first. I could now spend more time with Mom and could work at leisure. However, it is not as good as it looks like. At times it just feels you are busy without business, that you had been working, yet didn't achieve much. If you are evaluating working from home and are not sure of how to start, or you already do (then please review and add your views in comments) and feel like you were better off in the office, do read on. Remote work is great. But a physical office is better. So if you can, find yourself a co-working s

The economics of crypto investing

If you believe in the greater fool theory, there is no other market as speculative and volatile as the crypto market today. We are perhaps living in the biggest bubble of our times. I am not bullish on this market in particular. I am bullish on the mania. 90% of the cryptos we see today will crash. They are just tokens with no tangible value generation capability. However, I believe that the mania and euphoria will stay. Having said that, should one consider investing in this market? Certainly! The risk/reward is lovely, potential upsides and margins are huge and with 3-5% of your net worth, the bet on the mania is worth it. How does one choose where to invest? If you follow the stock markets, you are expected to do thorough Fundamental Analysis before investing. Expect the same for the crypto market. I invest in large caps. I invest in index funds. And I invest over and over again. Markets rise, always. Extrapolating the same strategy - invest in indices - the top 10 tokens by