This should have happened last year, but I ended up doing 3 changes at the same time which caused things to be more complicated than needed. This is what happened:
A common complaint that many people had for a long time was that uploading pictures was painfully slow, timing out in many occasions.
The problem was the way we handled them; and allowing to upload many files at once only made things N times slower. The processing for each of them was very naive.
There was a mishap on the last update regarding the signup process on the site that affects users that authenticate using their Reddit accounts.
Reddit does not provide third party sites the email address the user has on file. I think this is a fine security policy. The problem is that in the old signup flow we'd always ask users for an email address. In the new update since most authentications providers would give an email address we opted to remove that screen completely forgetting about Reddit.
Just had to do a server upgrade, our database server went up from 2GB to 4GB in ram and while at it also increased the swap partition from 512 to 2GB. Also did an security update of the packages on the server that was needed.
This really made a difference on the database performance. I still need to do a lot of tuning to our Postgres database, but on its current state it was swapping like crazy which caused lots of deadlocks and very poor performance. Lots of simple queries were taking over a second to complete.
This past week I did a rewrite on the listings in order to support searching by bra measurements. This made it just natural to be able to get listings notifications by this criteria as well.
This presented a number of problems.
It was 11 PM March 31 when I saw a tweet warning about April's fools and I realized that I had nothing prepared for the site so I needed to think of something quick to implement. My first idea was to turn the site black and white, but that would have required a lot of CSS and images editing to get going.
Then after some conversations about cup depth, the image of a face trying to get its nose to the deepest part of the cup came to my head and the idea of smelling the bra followed shortly after. It was something simple enough to implement.
Many events on the site have repercussion beyond the direct object where they acted up on and executing this repercussion can take some time. If we waited for all this to happen before showing the next page to the user, the site would be extremely slow.
These are some examples of such activities:
The last few weeks I've been working on improving the site's performance since beginning November last year it went to the floor.
There are still some pieces that I have not found completely but I spotted one of the biggest problems. It was template rendering. The Django views were returning too basic objects on the template's rendering context and too much logic was put on the templates, so they ended up doing many related queries (hidden in methods) or template tags.
It always bugged me that in order to render any page where the user is logged in I had to do a number of database queries to be able to render the notifications and main menu. Specifically: