When does a web-app become a large web-app?

[NickMNS]

I am currently building a web-app using Bottly.py. This is a "microframework" that is said to be good for small web-apps. It is similar to Flask. I like it because it is simple and easy to understand and I can easily get it to do what I want. This week I realized that I made a mistake in my design that is going to require some re-work and it has made me question whether my choice of framework is a good one. Things are manageable now, but as I add features I am afraid that I am going to reach a point where it will no longer be.

Other frameworks such a Pyramid and Django are said to be geared towards "large" web-apps. But what does that mean?
How is size measured?
Is it in lines of code?
Complexity?

I have little experience building web-apps so I really have no idea what things will look like when I am finally done.

I am using MongoDB as my backend, Apache for the server with MOD_WSGI. I currently have simple app running now, and I am expanding on this existing app.

[graeme_p]

When does a piece of string become a long piece of string?

I do not think that it large vs small is the right criterion. It is whether you need any of the capabilities built into a large frame work - because you would waste time reinventing the wheel if you use a smaller one.

Comparing Django to Bottle (because those are the two I have used). I would say:

1) If you need user authentication, use Django - it has a robust, customisable, built in authentication system (its in the contrib module, but it is still supported by the Django developers)
2) If you want a fast and easy backend admin, use Django.
3) if you want forms that save data entered to a database, use Django
etc.

A very simple application may meet these requirements: for example, a one page web app with a form that gathers data that is saved to a database (a survey form or similar, perhaps) would do so.

If in doubt about future requirements, use Django. I once used Bottle because I only needed to query databases and return JSON responses, and I then found I needed a lot more so the site now uses both and I am going to have to migrate the Bottle stuff to Django at some point in the future. You lose very little by using Django - you just have a lot of extra stuff available if you need it.

I would be reluctant to use Mongo with Django - but then, now that PostgreSQL has JSON, array and other field types that do most of what NoSQL databases can, there is less need to.

[NickMNS]

Graeme thanks for the reply, I really appreciate your input.

My situation is somewhat unique, in that I already have good part of my data in MongoDB database. Much of my code, the data analytics part of it (content generation) that occurs mostly off line and is not a direct part of the website, is written for JSON / MongoDB. So using a SQL ORMDB is out of the question.

The big problem with Django as I understand it is that authentication and backend admin functionalities will not work with MongoDB straight out of the box. So then what is the benefit?

I have another issue with Django. I may be wrong about this issue, as it is based on my very limited use of Django. It seems that it is heavily geared towards a basic CRM design pattern. That is log in, ask for a page, display a page, leave a comment, post, like etc... then leave. My site will require certain amount of data manipulation, that is it collects data input by the user performs some calculations and then fetches some corresponding data, merge it all together and display a page. It seems to that the more you move away from CRM/Blog design pattern the more the pre-built functionalities of Django start to fall apart. So why use it?

So I tend to agree it is not large vs small, and it isn't even simple vs complex. The question is, do the pre-built functionalities meet the needs?

Am I seeing this wrong? Why is Django so prevalent? Is it that most people are just sticking to this basic model. As I mentioned in my original post, I have little experience building web-apps, so I am concerned that I missing something, something that would be obvious to someone with a lot of experience, and that will cause me a lot of pain at some future point it in the process.

[graeme_p]

In that case you might find [django-mongodb-engine.readthedocs.io...] a good fit. There are some other approaches to doing this.

Without knowing the details of your application it is hard to comment in any detail, but Django provides user authentication, form processing, routing etc. which you still need. In general, even if the core of your app does not fit into Django's model, you are still likely to need it for large parts of your app.

Another alternative is to use Django to hand user authentication, form handling etc. keeping that data in an RDBMS and use Mongo directly from the code without any special integration with Django. I have done this when I needed to show data from both Solr and Postgres in a Django web app - Postgres is used through the Django ORM, Solr with a Python binding with no special support for Django. I may be doing something similar with a time series database soon (we have time series data that is processed using scikit-learn and the results displayed by a web app.).

If you would like to discuss the specifics or what you are trying to do, or of sites I have done that may be relevant, send me a message. It would be interesting to compare notes.

[NickMNS]

Thanks again Graeme. I am looking into django-mongodb-engine now. My intention was to use Cork for user authentication. So there is much to consider.

[graeme_p]

I have been doing some work on a large Flask project since my comments above. My feeling now is that Flask is slightly simpler for small projects, but on large projects you end up adding a bunch of components for things like authentication and form generation, so it actually ends up a lot more complex.

Bottle can be really very simple - if can be very nice for running something really simple - for example a REST API without authentication.

[explorador]

Difficult question. In my experience the web app gets too large when it takes me too long to recover from accidents, and this is very difficult to discuss because many times it's about design. By accidents I mean how many things this app depends on, and how many strings I have to tie when I move it from server to server. Example: on a large company decided to move sites to Wordpress, so many pages, so slow. Then the servers needed too many resources and when migrating, it was slow, painful. On my end my CMS (designed by me) was fast and didn't even use a Mysql database, I used flat files. Thousands of pages were moved on a tar file, expanded and it was working again.

It's difficult for me not to discuss this matter on a "technology or framework choice" central point of attention, and also design.

Nick: This week I realized that I made a mistake in my design that is going to require some re-work and it has made me question whether my choice of framework is a good one. Things are manageable now

I tried for years to explain this to clients when the project looks big. Any missing info or mistake can turn the project into an nightmare (more than being too big, it's about planning and design). And I've seen many frameworks promising flexibility and yada yada but at the end one change and it's just too much of an impact.

[graeme_p]

Which specific frameworks did not deliver flexibility? I have been using Django on a huge variety of sites for 10 years and have run into very few limitations, and the only one that proved problematic was the user model that was hard to customise (which was fixed a few years ago).

Wordpress is not a framework, it is a CMS. It is known to be slow. It makes typically tens of database requests per page and some plugins can lead to it making hundreds for some pages. It has no built in caching. A well designed site based a real framework would make FAR fewer queries and more efficient queries and would make caching fairly painless.

Moving a large commerical site to Wordpress without testing it can handle the load is incompetent. Did they even use a cache plugin? I dislike Wordpress and no longer use it except for my personal blog, but even I know you should use a cache plugin except for low traffic sites.