SQLite: The Database at the Edge of the Network with Dr. Richard Hipp

This point is echoed by the creator of SQLite, Dr Richard Hipp, in his excellent talk "SQLite: The Database at the Edge of the Network". [1] "SQLite doesn't compete with client-server databases, it competes with fopen" (I highly recommend to watch it, if only to get a sense of the smart and funny energy that Dr Richard Hipp puts off!) [1]: ▶ this video

I stumbled on this video presentation by Dr Hipp recently (may have been from another HN comment). I really enjoyed it. Probably because he seems so enthusiastic and passionate. SQLite: The Database at the Edge of the Network with Dr. Richard Hipp ▶ this video

This talk is really interesting too ▶ this video big industry projects (like the airline mentioned) probably contribute

> Hmmmm, there's a better talk by D. Richard Hipp around, but I'm not finding it atm. In the better talk he explains they (SQLite) are planning to be around for the 30 year timeframe you mention. Something to do with an airplane manufacturer looking for long term guarantees, but I don't remember the details. Here's the video[1] linked to from a tweet[2] linked to above.[3] [1]: ▶ this video [2]: link [3]: link

I am fairly certain that Dr. Hipp has discussed changes in SQLite that were desired by Microsoft for integration into Windows, which came to pass via a foundation membership. I believe that this was mentioned in the video below (I am not able to verify for now): ▶ this video It would be interesting to see what is required for an organization to negotiate non-trivial changes in SQLite.

It "is" (that's how the creator originally thought about it) but it also isn't really prescribed as the only interpretation ▶ this video Though I'll also note the "lite" crowd often interpret the "L" as double duty/shared (like the "m" in "dramedy" isn't necessarily interpreted as solely part of "drama" or "comedy").

Sorry. It's just a joke. I saw the original in Dr. Richard Hipp's "SQLite: The Database at the Edge of the Network" and thought it was pretty funny. ▶ this video I added a "(humor)" tag at the end.

The name SQLite isn't Sequel-lite. It's S.Q.L.-ite akin to graphite or Levite. It means a thing of SQLy stuff. The creator says it like Escue Ell-ite emphasis on the Es- and -ite. [0] ▶ this video

Either way, the name SQLite is not SQL-lite but SQL-ite (think graphite). SQLite is an element of SQLey stuff. Source: ▶ this video I'd link the exact timestamp, but then you might watch less than the entire talk, which would be unfortunate.

DuckDB looks very interesting and I'm quite excited to examine it more closely in the next few days! I just wanted to add to the discussion that an unchanging file format, or at least a backwards compatible one, is a key feature of sqlite. See for example Richard Hipp's comments here [1] (I think he also mentioned earlier in the talk that the file format has become a limiting factor now in terms of some of the refactoring that they can do). The file format therefore seems likely to be a major factor in the long term success of this project and I am glad to see that you are taking your time before settling on any architecture here. Given that you are targeting the data science and analytics space, what are your plans for integration with arrow and the feather file format? From a purely user/developer perspective, arrow's aim of shared memory data structures across different analytics tools, seems like a great goal. I know Wes McKinney and Ursa Labs have also spent quite some time at the file storage part of this, see for example the Feather V2 announcement [2]. What are your thoughts on the tradeoffs they considered and how do you see the requirements of DuckDB in relation to theirs? From the Carnegie Mellon DuckDB talk [3], I saw that you already have a zero-copy reader to the pandas memory data structures, so the vision I have is that DuckDB could be the universal SQL interface to arrow datasets which can then also be shared with more complex ML models. Is that something tha

I've been reading through many of the other pages on the SQLite website. I didn't realize that SQLite is developed by a company (Hwaci) and funded by selling support, licenses, and testing services. I had always assumed it was an open source project backed by a non-profit, similar to the Apache Software Foundation. This is another very interesting example of an open-source business. I would be interested to learn more about how the Hwaci company operates (revenue, number of employees, etc.). I find this very interesting: > We are a 100% engineering company. There is no sales staff. Our goal is to provide outstanding service and honest advice without spin or sales-talk. They list some "$8K-50K/year" and "$85k" price tags directly on their "Pro Support" webpage. These would usually be behind a "Schedule a Call" or "Get Quote" button. I've been thinking about doing something similar with my own on-premise licenses and support contracts. I'm not very good at sales and I don't really want to hire a sales team, so I'd be interested to know how this worked out for them. I also liked this sentence, which is very similar to Basecamp's philosophy (and both companies were started around the same time - 1999 vs 2000): > Hwaci intends to continue operating in its current form, and at roughly its current size until at least the year 2050. It's interesting to think that SQLite could have raised money and grown into a billion-dollar public company with thousands of employees. I'm going to li

They actually have to test to that degree to follow aviation standards (DO-178b [0]) because they're used in aviation equipment. Dr. Hipp said he started really following it when Android came out and included SQLite and suddenly there were 200M mobile SQLite users finding edge cases: ▶ this video Lightly edited transcript here: > It made a huge difference. That that was when Android was just kicking off. In fact Android might not have been publicly announced, but we had been called in to help with getting Android going with SQLite. [Actually], they had been publicly announced and there were a bunch of Android phones out and we were getting flooded with problems coming in from Android. > I mean it worked great in the lab it worked great in all the testing and then [...] you give it to 200 million people and let them start clicking on their phone all day and suddenly bugs come up. And this is a big problem for us. > So I started doing following this DO-178b process and it took a good solid year to get us there. Good solid year of 12 hour days, six days a week, I mean we really really pushed but we got it there. And you know, once we got SQLite to the point where it was at that DO-178b level, standard, we still get bugs but you know they're very manageable. They're infrequent and they don't affect nearly as many people. > So it's been a huge huge thing. If you're writing an application deal ones, you know a website, a DO-178b/a is way overkill, okay? It's just because it's very expen