HN Theater @HNTheaterMonth

> For intance, when is it ever OK to let your client write directly in the database, even for their own data? If you're going to pass their calls through a deny and allow call, why not just expose RPCs to the client that will handle any writing?

If you have a client-side mirror of your server-side database whose changes are monitored by the templating engine, you can execute what Meteor calls "latency compensation", where changes users make are instantly reflected in the UI and synced to the server in the background. This is a central feature of Meteor. More here: https://www.meteor.com/full-stack-db-drivers and in this Youtube video: https://www.youtube.com/watch?v=tqLbodVH3dw

> It seems that a lot of the light versality you have at prototyping time is lost whenever you have to get it production ready. There is definitely some value in light prototypes, but it looks like a real pain to go from prototype to production.

It's actually very easy to go from prototype to production; you just remove some convenience packages (autopublish and insecure), set up your security rules, do some basic performance tuning (eg. don't publish entire documents if you only need 3 fields, which is a standard practice for any app) and that's about it. Of course the definition of "production" varies per project, but after shipping over 20 Meteor apps since 2012 these are the most common things I tend to do.

> Likewise, I'm confused about how to get around the limitations of mongodb. Say you run a store with a finite inventory, how do you handle concurrent purchases? How about a website to enroll into classes? How about a webforum which can have exactly 5 administrators?

You can express most of these in code. Yes, in relational databases you can express constraints in the database. But then you're having a NoSQL vs SQL debate, which is sort of out of scope of this discussion.

⬐ murbard2

Sure you remove the autopublish package, but then your app stops working. You need to explicitely publish what you want published, you need to explicitely check and validate every input from the client... and that's fine, but then what exactly have you gained from using meteor over, say, socket.io or autobahn? Fast prototyping, yes, you've definitely gained that, but have you gained anything else? I'm genuinely asking, not making a rhetorical point.

Since meteor integrates very tightly with mongodb, a nosql database, I don't see how the limitations of nosql are out of the scope of the discussion? I also don't know what you mean by "you express most of these in code". The problem isn't writing a piece of code that expresses that constraint... the problem is doing so without introducing a race condition!

⬐ aliakhtar

> I also don't know what you mean by "you express most of these in code". The problem isn't writing a piece of code that expresses that constraint... the problem is doing so without introducing a race condition!

I agree, and this also exposes a weakness of javascript as a backend language. If you use Java/Scala, you have multithreading support and can sync the threads + concurrently verify that an item isn't being purchased more than X times. Which makes this problem solvable even with a NoSQL database. Javascript is single threaded though, which makes this problem a lot harder and I don't know how you'd solve it without an RDBMS with transactional support.

⬐ AlisdairO

> If you use Java/Scala, you have multithreading support and can sync the threads + concurrently verify that an item isn't being purchased more than X times.

That does cause you to have a hard limit of one application server for the app, which can be a pretty big deal...

⬐ aliakhtar

No, you could have multiple servers talking to each other via something like ZooKeeper. But individually, each node/server would use multithreading to keep track of its state.

⬐ AlisdairO

I guess, but that's some monumental hackery to overcome something that works perfectly fine in an RDBMS :-)

⬐ aliakhtar

Until the time comes when you need to scale out your RDBMS and need to shard it. IMO that's a bigger pain ;).

Plus if you have multiple servers, they will most likely need to pass messages for other things too.

⬐ AlisdairO

I dunno - you've presumably got to implement code to make sure that other servers get unlocked when your locking app server goes down, for example. Personally I try to avoid implementing ad-hoc distributed systems if I can avoid it. Having to shard is pretty damn crappy, but under the circumstances I'd favour it.

Further, I tend to think that if your application is heavy enough to outgrow a reasonably fat db server with (say) 500+GB of RAM, strategies like reimplementing consistent-db-esque locking in an app server + zookeeper setup are also likely to hit some pretty weird performance issues.

⬐ aliakhtar

> you've presumably got to implement code to make sure that other servers get unlocked when your locking app server goes down

Don't see why you'd need locking. Node A receives a request to buy an item, A asks nodes B, C, and D to confirm that the item is in stock in each node's internal state. If all give the OK, A lets the purchase go through. If node B goes down, then A just asks C and D. I don't see the need for locking or a master/slave.

> a reasonably fat db server with (say) 500+GB of RAM

You have a single point of failure with a single, monolithic server. If its hd dies, you will be in serious trouble restoring terabytes of db.

⬐ AlisdairO

EDIT: On rereading it occurs to me that I'm getting caught up in poking holes here. The proposed ZooKeeper solution will likely work fine in the simple case, if used with care and attention. It gives me the willies a bit (because I don't think it's a good idea to force users to separate locking from the DB level), but as long as the use case is restricted to compare + set on a single document, there's little reason to think that there's much wrong with it. Apologies if I come across as belligerent.

-----

> Don't see why you'd need locking. Node A receives a request to buy an item, A asks nodes B, C, and D to confirm that the item is in stock in each node's internal state. If all give the OK, A lets the purchase go through. If node B goes down, then A just asks C and D. I don't see the need for locking or a master/slave.

Presumably in order to ensure that B, C, and D can consistently check the object's state, nothing else can be allowed to write to it at that time - A effectively has to lock the object using zookeeper. You then have to make sure that the object gets unlocked if A goes down after locking it, or most particularly if A hangs, keeping the zookeeper session open? Otherwise your other app servers could get caught in a very long or infinite lock wait.

You've also got to keep track of the rest of your code, and make sure it never alters these same objects without locking. I'm assuming here that you only distributed-lock objects when you absolutely have to, in order to improve performance: otherwise you're going to start hitting some of the issues that make it hard to cluster relational DBs with decent performance (modulo some coarser lock granularity for a document-based system).

As you add complexity to this approach (say, perhaps you need to work on two objects at once), you also start having to think about other problems like deadlocking, or what happens if your zookeeper session fails part way through your work, where you've made half of the changes you want to make, and you can't easily roll back. This is all fine if you have programmers who are competent to think about these kinds of problems, but they're typically pretty expensive - and my experience is that most people just don't really bother to think about it.

> You have a single point of failure with a single, monolithic server. If its hd dies, you will be in serious trouble restoring terabytes of db.

Any system of value will of course have a replica, meaning you're perhaps more likely to be worried about network problems than the server falling over. Of course this is still less reliable than a perfectly implemented multi-master distributed system, but it's also hugely easier to use correctly - and the likelihood of failure of a single node is actually very low. Obviously if you literally cannot afford any downtime, maybe you go with a distributed system (or follow the banks and use mainframes..), but businesses that will be killed by network blips are certainly in the minority.

⬐ dsyko

To answer your first question, there are a number of attractive advantages to using Meteor over say socket.io or autobahn on top of Node. One major one would be a consistent API for accessing your data on the Client and Server. I would argue that using Javascript on both backend and frontend with Node boosts a developer's productivity by reducing language/context switching. In the same vein accessing data using the Mongo query api on both backend and front end with Meteor's MiniMongo improves development. Also Meteor comes with built-in latency compensation when using either RPCs (with stubs on the client) or Minimongo operations. Latency compensation is a non-trivial problem which would need to be implemented by the developer when using something like socket.io. You also gain access to a growing list of great packages that implement things like user accounts which you would otherwise need to write yourself or wire together from various projects.

As for your second question. There are certainty use cases where MongoDB isn't appropriate, and support for other databases is on Meteor's roadmap. For now there are some community created packages for interfacing directly with SQL databases in place of MongoDB, but there are a number of issues with them. You could also pipe data in and out of an SQL database inside RPCs, however the data would not be updated in 'real-time' to clients. I anticipate SQL databases will get some good support options in time.

⬐ karanbhangui

Looks like this is from a CS course at UC Berkeley on cutting edge web technologies: http://inst.eecs.berkeley.edu/~cs294-101/sp15/

Good talk from Pete Hunt on React as well. More talks from others will be coming in the near future.

⬐ bsaul

Very interesting talk, but being in the process of building a similar architecure at the moment, i think the person dismiss OT and other kind of merging technics a bit too fast. "Latest to write wins" isn't manageable in every case , such as google doc, or github. You sometimes needs locking and atomic complex updates for which a stack only built on top of mongo and pojo, and no other knowledge of the relations between data parts may be insufficient.

But i sure hope they'll prove me wrong and solve the problem in a generic way.

⬐ mizzao

You can always use an existing OT stack within Meteor, such as this package I wrote integrating ShareJS: https://github.com/mizzao/meteor-sharejs

⬐ primigenus

Meteor should publish more videos like this explaining the fundamentals. I feel like a lot of people miss the thinking behind the platform and end up comparing Meteor to other libraries and frameworks they already know, such as Ember, Angular or React, when Meteor is actually something completely different.

For instance, if you visit the "projects" page on the Meteor website (https://www.meteor.com/projects), you're faced with the following introductory paragraph:

> The Meteor Project is a little bit like the Apache Foundation or the Free Software Foundation in that it is an umbrella organization that stewards the development of a set of open source projects. This page has information on each of the subprojects currently being sponsored by the Meteor Project.

It would be interesting to hear more about their philosophy on simultaneously building an organisation like this and designing a software platform.

⬐ mizzao

It seems that the ability to tell what is fundamentally different about one framework versus another, instead of just treating all frameworks as interchangeable keywords, is what distinguishes some good engineers from bad :)

⬐ mrgordon

Agreed, I'm lucky enough to know the founders personally and it is quite inspiring to hear where they're going and why they decided this was what needed to be built.